Adaptive surgical system control according to surgical plume cloud characteristics

Abstract

Various control systems for controlling surgical systems based on the characteristics of detected aerosol cloud are disclosed. The control system can adaptively control various surgical devices (such as surgical instruments and fume extractors) according to changes in the configuration or state of surgical smoke cloud or another aerosol detected at the surgical site.

Description

Background Technology

[0001] Surgical systems are often integrated with imaging systems that allow clinicians to view the surgical site and / or one or more parts thereof on one or more monitors, such as a monitor. The monitors can be local to the operating room and / or remote. Imaging systems may include viewing mirrors with cameras that view the surgical site and transmit the view to a monitor accessible to the clinician. Viewing mirrors include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, colonoscopes, esophagogastric-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngoscopes-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and external endoscopes. Imaging systems may be limited by the information they can identify and / or convey to the clinician. For example, some imaging systems may not be able to identify certain hidden structures, physical contours, and / or dimensions in three-dimensional space during surgery. Additionally, some imaging systems may not be able to transmit and / or convey certain information to the clinician during surgery. Summary of the Invention

[0002] In one general aspect, a control system for a surgical system is disclosed. The control system includes an imaging system and control circuitry coupled to the imaging system. The imaging system includes a transmitter configured to emit multispectral electromagnetic radiation (EMR). The control circuitry is configured to detect airborne particles via the emitted EMR through the image sensor, characterize the boundary of a particle cloud containing the detected airborne particles, and adjust control parameters of the surgical system based on the characteristics of the particle cloud boundary.

[0003] In another general aspect, a control system for a surgical system is disclosed. The control system includes an imaging system and control circuitry coupled to the imaging system. The imaging system includes a multispectral electromagnetic radiation (EMR) source and an image sensor. The control circuitry is configured to detect EMR reflected from particles in an aerosol and received by the image sensor, characterize the configuration of the aerosol containing the detected particles, and adjust at least one of a first control parameter or a second control parameter of the surgical system based on the characterized configuration of the aerosol.

[0004] In yet another general aspect, a method for controlling a surgical system is disclosed. The surgical system includes an imaging system comprising a transmitter configured to emit multispectral electromagnetic radiation (EMR) and an image sensor. The method includes the steps of: detecting airborne particles based on the emitted EMR via the image sensor, characterizing the boundary of a particle cloud containing the detected airborne particles, and adjusting control parameters of the surgical system according to the characteristics of the particle cloud boundary. Attached Figure Description

[0005] The novel features of various aspects are specifically set forth in the appended claims. However, the described aspects relating to both the organization and the method of operation are best understood by referring to the following description in conjunction with the accompanying drawings, wherein:

[0006] Figure 1 This is a schematic diagram of a surgical visualization system including an imaging device and a surgical apparatus according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify key structures beneath the surface of tissue.

[0007] Figure 2 It is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0008] Figure 2A A control circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0009] Figure 2B A combinational logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0010] Figure 2C A timing logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0011] Figure 3 It is described in accordance with at least one aspect of this disclosure. Figure 1 Triangulation is performed between the surgical device, imaging device, and key structure to determine the depth d of the key structure below the tissue surface. A A schematic diagram.

[0012] Figure 4 This is a schematic diagram of a surgical visualization system configured to identify critical structures beneath the tissue surface according to at least one aspect of this disclosure, wherein the surgical visualization system includes methods for determining the depth d of the critical structures beneath the tissue surface. A A pulsed light source.

[0013] Figure 5 This is a schematic diagram of a surgical visualization system including an imaging device and a surgical apparatus according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify key structures beneath the surface of tissue.

[0014] Figure 6 This is a schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, wherein the surgical visualization system is configured to identify key structures embedded within tissue.

[0015] Figure 7A and Figure 7B It is based on at least one aspect of this disclosure. Figure 6 A view of the key structure captured by a 3D camera, in which Figure 7A It is a view from the left lens of a 3D camera, and Figure 7B This is a view from the right lens of a 3D camera.

[0016] Figure 8 It is based on at least one aspect of this disclosure Figure 6 A schematic diagram of a surgical visualization system, in which the camera-key structure distance d from the 3D camera to the key structure can be determined. w .

[0017] Figure 9 This is a schematic diagram of a surgical visualization system that utilizes two cameras to determine the orientation of embedded key structures, according to at least one aspect of this disclosure.

[0018] Figure 10A This is a schematic diagram of a surgical visualization system utilizing a camera according to at least one aspect of this disclosure, the camera moving axially between multiple known orientations to determine the orientation of an embedded key structure.

[0019] Figure 10B It is based on at least one aspect of this disclosure Figure 10A A schematic diagram of a surgical visualization system in which a camera moves axially and rotationally between multiple known orientations to determine the orientation of embedded key structures.

[0020] Figure 11 It is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0021] Figure 12 This is a schematic diagram of a structured light source for a surgical visualization system according to at least one aspect of this disclosure.

[0022] Figure 13A It is a graph of the absorption coefficients of various biological materials at different wavelengths according to at least one aspect of this disclosure.

[0023] Figure 13B It is a schematic diagram of visualizing anatomical structures through a spectral surgical visualization system according to at least one aspect of this disclosure.

[0024] Figures 13C to 13E Exemplary hyperspectral identification features for distinguishing anatomical structures from obscuring objects, according to at least one aspect of this disclosure, are described, wherein Figure 13C It is a graphical representation of the characteristics of the ureter and its covering. Figure 13D It is a graphical representation of arterial features and obstructions, and Figure 13E It is a graphical representation of neural features and obscuring objects.

[0025] Figure 14 This is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distances to critical anatomical structures according to at least one aspect of this disclosure, the time-of-flight measurement system including a transmitter and a receiver (sensor) positioned on a common device.

[0026] Figure 15 It is based on at least one aspect of this disclosure Figure 14 A schematic diagram of the transmitted wave, received wave, and the delay between the transmitted and received waves of a NIR time-of-flight measurement system.

[0027] Figure 16 A NIR time-of-flight measurement system configured to sense distances from different structures, according to at least one aspect of the present disclosure, is shown. The time-of-flight measurement system includes a transmitter (transmitter) and a receiver (sensor) on separate devices.

[0028] Figure 17 This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of this disclosure.

[0029] Figure 18 It is a surgical system for performing surgical procedures in an operating room, according to at least one aspect of this disclosure.

[0030] Figure 19 An interactive surgical system implemented by a computer according to at least one aspect of the present disclosure is shown.

[0031] Figure 20 A diagram of a situational awareness surgical system according to at least one aspect of this disclosure is shown.

[0032] Figure 21 A timeline depicting the situational awareness of a hub according to at least one aspect of this disclosure is shown.

[0033] Figure 22 It is a diagram of a surgical system during the performance of a surgical procedure according to at least one aspect of this disclosure.

[0034] Figure 23 This is a diagram of an imaging apparatus facing multiple obstructions according to at least one aspect of this disclosure.

[0035] Figure 24 This is a logic flowchart of a process for generating a fused image using a multispectral EMR source according to at least one aspect of this disclosure.

[0036] Figure 25 It is a graph of a fused image generated from a multispectral EMR source according to at least one aspect of this disclosure.

[0037] Figure 26 This is a logical flowchart of a process for generating a fused image using multiple image frames according to at least one aspect of this disclosure.

[0038] Figure 27 It is a diagram of a series of image frames according to at least one aspect of this disclosure.

[0039] Figure 28 It is a diagram of a fused image according to at least one aspect of this disclosure.

[0040] Figure 29 It is a graph of a fused image that is visualized to the user in accordance with at least one aspect of this disclosure.

[0041] Figure 30 This is a schematic diagram of a surgical instrument according to at least one aspect of this disclosure.

[0042] Figure 31 It is a logic flowchart of a process for controlling a surgical system based on multiple sensing parameters according to at least one aspect of this disclosure.

[0043] Figure 32 This is a graph of polarized EMR sources for detecting different particle types according to at least one aspect of this disclosure.

[0044] Figure 33A This is a logic flowchart of a process for controlling a surgical system based on the detected particle type according to at least one aspect of the present invention.

[0045] Figure 33B This is a logic flowchart of a process for controlling a surgical system based on the type of particles detected within a defined distance gating according to at least one aspect of the present invention.

[0046] Figure 34A It is a pixel array of an image sensor for detecting airborne particles according to at least one aspect of this disclosure.

[0047] Figure 34B The detection based on at least one aspect of this disclosure has been obtained from Figure 34A The image sensor's pixel array shows the positional movement of airborne particles.

[0048] Figure 34C It is a pixel array of an image sensor according to at least one aspect of this disclosure, which indicates Figure 34B The generalized motion vector of the particle shown.

[0049] Figure 35 The diagram shows at least one aspect corresponding to the present disclosure. Figures 34A-34C Changes in the state of airborne particulate clouds.

[0050] Figure 36 This is a diagram of a surgical system during the execution of a surgical procedure that generates a cloud of microparticles, according to at least one aspect of this disclosure.

[0051] Figure 37 It is a logical flowchart of the process of controlling a surgical system based on the characteristics of a microparticle cloud, according to at least one aspect of this disclosure.

[0052] Figure 38 The present disclosure illustrates a series of graphs showing the adjustment of control parameters based on the characteristics of a particulate cloud, according to at least one aspect of the present disclosure. Detailed Implementation

[0053] The applicant of this application holds the following U.S. patent applications, each of which is incorporated herein by reference in its entirety:

[0054] • The agent's case file number is END9228USNP1 / 190580-1M, and its title is "METHOD OF USING IMAGINGDEVICES IN SURGERY";

[0055] • The agent's case file number is END9227USNP1 / 190579-1, and its name is "ADAPTIVE VISUALIZATIONBY A SURGICAL SYSTEM";

[0056] • The agent's case file number is END9226USNP1 / 190578-1, and its title is "SURGICAL SYSTEM CONTROLBASED ON MULTIPLE SENSED PARAMETERS";

[0057] • The agent's case file number is END9225USNP1 / 190577-1, and its title is "ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS";

[0058] • The agent's case file number is END9223USNP1 / 190575-1, and its title is "SURGICAL SYSTEMSCORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA";

[0059] • The agent's case file number is END9222USNP1 / 190574-1, and its title is "SURGICAL SYSTEMS FORGENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLINGIDENTIFIED";

[0060] • The agent's case file number is END9221USNP1 / 190573-1, and its title is "SURGICAL SYSTEM FOROVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN";

[0061] • The agent's case file number is END9220USNP1 / 190572-1, and its title is "SURGICAL SYSTEMS FORPROPOSING AND CORROBORATING ORGAN PORTION REMOVALS";

[0062] • The agent's case file number is END9219USNP1 / 190571-1, and its title is "SYSTEM AND METHOD FORDETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE";

[0063] • The agent's case file number is END9218USNP1 / 190570-1, and its name is "VISUALIZATION SYSTEMSUSING STRUCTURED LIGHT";

[0064] • The agent's case file number is END9217USNP1 / 190569-1, and its name is "DYNAMIC SURGICALVISUALIZATION SYSTEMS"; and

[0065] • The agent's case file number is END9216USNP1 / 190568-1, and its title is "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM".

[0066] The applicant of this application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety:

[0067] • U.S. Patent Application Serial No. 16 / 354,417, entitled “INPUT CONTROLS FOR ROBOTICSURGERY”;

[0068] • U.S. Patent Application Serial No. 16 / 354,420, entitled “DUAL MODE CONTROLS FORROBOTIC SURGERY”;

[0069] • U.S. Patent Application Serial No. 16 / 354,422, entitled “MOTION CAPTURE CONTROLS FORROBOTIC SURGERY”;

[0070] • U.S. Patent Application Serial No. 16 / 354,440, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY”;

[0071] • U.S. Patent Application Serial No. 16 / 354,444, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TO TISSUE”;

[0072] • U.S. Patent Application Serial No. 16 / 354,454, entitled “ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS”;

[0073] • U.S. Patent Application Serial No. 16 / 354,461, entitled “SELECTABLE VARIABLE RESPONSEOF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS”;

[0074] • U.S. Patent Application Serial No. 16 / 354,470, entitled “SEGMENTED CONTROL INPUTS FORSURGICAL ROBOTIC SYSTEMS”;

[0075] • U.S. Patent Application Serial No. 16 / 354,474, entitled “ROBOTIC SURGICAL CONTROLSHAVING FEEDBACK CAPABILITIES”;

[0076] • U.S. Patent Application Serial No. 16 / 354,478, entitled “ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK”; and

[0077] • U.S. Patent Application Serial No. 16 / 354,481, entitled “JAW COORDINATION OF ROBOTICSURGICAL CONTROLS”.

[0078] The applicant of this application also owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety:

[0079] • U.S. Patent Application Serial No. 16 / 128,179, entitled “SURGICAL VISUALIZATION PLATFORM”;

[0080] • U.S. Patent Application Serial No. 16 / 128,180, entitled “CONTROLLING AN EMITTERASSEMBLY PULSE SEQUENCE”;

[0081] • U.S. Patent Application Serial No. 16 / 128,198, entitled “SINGULAR EMR SOURCE EMITTERASSEMBLY”;

[0082] • U.S. Patent Application Serial No. 16 / 128,207, entitled “COMBINATION EMITTER AND CAMERA ASSEMBLY”;

[0083] • U.S. Patent Application Serial No. 16 / 128,176, entitled “SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES”;

[0084] • U.S. Patent Application Serial No. 16 / 128,187, entitled “SURGICAL VISUALIZATION OF MULTIPLE TARGETS”;

[0085] • U.S. Patent Application Serial No. 16 / 128,192, entitled “VISUALIZATION OF SURGICALDEVICES”;

[0086] • U.S. Patent Application Serial No. 16 / 128,163, entitled “OPERATIVE COMMUNICATION OF LIGHT”;

[0087] • U.S. Patent Application Serial No. 16 / 128,197, entitled “ROBOTIC LIGHT PROJECTIONTOOLS”;

[0088] • U.S. Patent Application Serial No. 16 / 128,164, entitled “SURGICAL VISUALIZATIONFEEDBACK SYSTEM”;

[0089] • U.S. Patent Application Serial No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MONITORING”;

[0090] • U.S. Patent Application Serial No. 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA”;

[0091] • U.S. Patent Application Serial No. 16 / 128,170, entitled “ROBOTICALLY-ASSISTED SURGICALSUTURING SYSTEMS”;

[0092] • U.S. Patent Application Serial No. 16 / 128,183, entitled “SAFETY LOGIC FOR SURGICALSUTURING SYSTEMS”;

[0093] • U.S. Patent Application Serial No. 16 / 128,172, entitled “ROBOTIC SYSTEM WITH SEPARATEPHOTOACOUSTIC RECEIVER”; and

[0094] • US Patent Application Serial No. 16 / 128,185, entitled “FORCE SENSOR THROUGHSTRUCTURED LIGHT DEFLECTION”.

[0095] The applicant of this application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0096] • U.S. Patent Application Serial No. 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201111;

[0097] • U.S. Patent Application Serial No. 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTSFOR ROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201142;

[0098] • U.S. Patent Application Serial No. 15 / 940,711, entitled “SENSING ARRANGEMENTS FORROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201120; and

[0099] • U.S. Patent Application Serial No. 15 / 940,722, entitled “Characterization of Tissueir Reggaerities Through the Use of Mono-Chromatic Light Refraction”, now U.S. Patent Application Publication No. 2019 / 0200905.

[0100] The applicant of this patent application owns the following U.S. patent applications filed on December 4, 2018, the disclosure of each of which is incorporated herein by reference in its entirety:

[0101] • U.S. Patent Application Serial No. 16 / 209,395, entitled “METHOD OF HUB COMMUNICATION”, now U.S. Patent Application Publication No. 2019 / 0201136;

[0102] • U.S. Patent Application Serial No. 16 / 209,403, entitled “METHOD OF CLOUD BASED DATAANALYTICS FOR USE WITH THE HUB”, now U.S. Patent Application Publication No. 2019 / 0206569;

[0103] • U.S. Patent Application Serial No. 16 / 209,407, entitled “METHOD OF ROBOTIC HUBCOMMUNICATION, DETECTION, AND CONTROL”, now U.S. Patent Application Publication No. 2019 / 0201137;

[0104] • U.S. Patent Application Serial No. 16 / 209,416, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS”, now U.S. Patent Application Publication No. 2019 / 0206562;

[0105] • U.S. Patent Application Serial No. 16 / 209,423, entitled “METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THETISSUE WITHIN THE JAWS”, now U.S. Patent Application Publication No. 2019 / 0200981;

[0106] • U.S. Patent Application Serial No. 16 / 209,427, entitled “METHOD OF USING REINFORCEDFLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIOFREQUENCY DEVICES”, now U.S. Patent Application Publication No. 2019 / 0208641;

[0107] • U.S. Patent Application Serial No. 16 / 209,433, entitled “METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THESENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB”, now U.S. Patent Application Publication No. 2019 / 0201594;

[0108] • U.S. Patent Application Serial No. 16 / 209,447, entitled “METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB”, now U.S. Patent Application Publication No. 2019 / 0201045;

[0109] • U.S. Patent Application Serial No. 16 / 209,453, entitled “METHOD FOR CONTROLLING SMARTENERGY DEVICES”, now U.S. Patent Application Publication No. 2019 / 0201046;

[0110] • U.S. Patent Application Serial No. 16 / 209,458, entitled “METHOD FOR SMART ENERGYDEVICE INFRASTRUCTURE”, now U.S. Patent Application Publication No. 2019 / 0201047;

[0111] • U.S. Patent Application Serial No. 16 / 209,465, entitled “METHOD FOR ADAPTIVE CONTROLSCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION”, now U.S. Patent Application Publication No. 2019 / 0206563;

[0112] • U.S. Patent Application Serial No. 16 / 209,478, entitled “METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE”, now U.S. Patent Application Publication No. 2019 / 0104919;

[0113] • U.S. Patent Application Serial No. 16 / 209,490, entitled “METHOD FOR FACILITY DATACOLLECTION AND INTERPRETATION”, now U.S. Patent Application Publication No. 2019 / 0206564; and

[0114] • U.S. Patent Application Serial No. 16 / 209,491, entitled “METHOD FOR CIRCULAR STAPLERCONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS”, now U.S. Patent Application Publication No. 2019 / 0200998.

[0115] Before detailing the various aspects of the surgical visualization platform, it should be noted that the illustrative examples are not limited in application or use to the details of the construction and arrangement of the components shown in the accompanying drawings and specifications. The illustrative examples may be implemented or incorporated in other aspects, variations, and modifications, and may be practiced or performed in various ways. Furthermore, unless otherwise specified, the terminology and expressions used herein are chosen for the convenience of the reader in describing the illustrative examples and are not intended to be restrictive. Moreover, it should be understood that one or more of the aspects, expressions, and / or examples described below may be combined with any one or more of the other aspects, expressions, and / or examples described below.

[0116] Surgical visualization system

[0117] This disclosure relates to a surgical visualization platform that utilizes "digital surgery" to obtain additional information about a patient's anatomy and / or surgical procedures. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a helpful manner. For example, various aspects of this disclosure provide improved visualization of a patient's anatomy and / or surgical procedures.

[0118] "Digital surgery" can encompass robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and much more. While the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, they are not limited to such use. In some cases, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0119] In some cases, surgical systems incorporating surgical visualization platforms can enable intelligent anatomy to identify and avoid critical structures. Critical structures include anatomical structures such as the ureter, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other cases, critical structures can be, for example, foreign structures within the anatomical field, such as surgical devices, surgical fasteners, clamps, pins, probes, bands, and / or plates. Critical structures can be determined based on different patients and / or different surgical procedures. Exemplary critical structures are also described herein. For example, intelligent anatomy techniques can provide intraoperative guidance for improved anatomy and / or enable intelligent decision-making using critical anatomical structure detection and avoidance techniques.

[0120] Surgical systems incorporating surgical visualization platforms can also enable intelligent anastomosis techniques, which utilize improved workflows to provide more consistent anastomosis at optimal locations. The various surgical visualization platforms and procedures described herein can also be used to improve cancer localization techniques. For example, cancer localization techniques can identify and track the location, orientation, and boundaries of cancer. In some cases, cancer localization techniques can compensate for movement of instruments, the patient, and / or the patient's anatomy during surgery to provide clinicians with guidance back to the point of interest.

[0121] In certain aspects of this disclosure, surgical visualization platforms can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly when dissecting tissues and / or placing suture devices. Some of the tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. Regarding lymph node diagnosis and mapping, surgical visualization platforms can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0122] During surgery, information available to clinicians via the naked eye and / or imaging systems can provide an incomplete view of the surgical site. For example, certain structures (such as those embedded or buried within organs) may be at least partially concealed or hidden from view. Additionally, certain dimensions and / or relative distances may be difficult to detect using existing sensor systems and / or difficult to perceive with the naked eye. Furthermore, some structures may be mobile preoperatively (e.g., before surgery but after preoperative scanning) and / or intraoperatively. In such cases, clinicians may be unable to accurately pinpoint the location of critical structures intraoperatively.

[0123] Clinicians' decision-making processes can be hampered when the location of critical structures is uncertain and / or when the proximity of critical structures to surgical instruments is unknown. For example, clinicians may avoid certain areas to prevent accidental dissection of critical structures; however, the avoided areas may be unnecessarily large and / or at least partially misaligned. Due to uncertainty and / or excessive caution, clinicians may be unable to access certain desired areas. For instance, excessive caution may lead clinicians to leave portions of tumor and / or other unwanted tissue in an attempt to avoid critical structures, even if the critical structures are not in that particular area and / or will not be negatively affected by a clinician working in that particular area. In some cases, surgical outcomes can be improved by increasing knowledge and / or certainty, which can enable surgeons to be more accurate in specific anatomical areas and, in other cases, make surgeons less conservative / more aggressive.

[0124] In various aspects, this disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, this disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what clinicians can see with the "naked eye" and / or beyond what imaging systems can identify and / or convey to clinicians. Various surgical visualization systems can enhance and strengthen what clinicians can know before tissue treatment (e.g., dissection) and thus improve outcomes in a variety of situations.

[0125] For example, a visualization system may include a first light emitter configured to emit multiple spectral waves, a second light emitter configured to emit light patterns, and one or more receivers or sensors configured to detect visible light, molecular responses to spectral waves (spectral imaging), and / or light patterns. It should be noted that throughout the entire disclosure below, unless specifically mentioned as visible light, any reference to “light” may include photons in the visible and / or invisible portions of electromagnetic radiation (EMR) or the EMR wavelength spectrum. Surgical visualization systems may also include an imaging system and control circuitry that communicates signals with the receiver and the imaging system. Based on the output from the receiver, the control circuitry may determine a geometric surface mapping (i.e., three-dimensional surface topography) of the visible surface at the surgical site and one or more distances relative to the surgical site. In some cases, the control circuitry may determine one or more distances to at least partially concealed structures. Furthermore, the imaging system may communicate the geometric surface mapping and one or more distances to the clinician. In such cases, the enhanced view of the surgical site provided to the clinician can represent concealed structures within the relevant environment of the surgical site. For example, an imaging system can virtually enhance occult structures on a geometric surface mapping of occult and / or obstructing tissue, similar to lines drawn on the ground to indicate practical lines beneath the surface. Additionally or alternatively, the imaging system can convey the proximity of one or more surgical instruments to visible obstructing tissue and / or to at least partially occulted structures, and / or the depth of the occult structure beneath the visible surface of the obstructing tissue. For example, a visualization system can determine the distance to an enhancing line relative to the surface of visible tissue and convey that distance to the imaging system.

[0126] In various aspects of this disclosure, surgical visualization systems for intraoperative identification and avoidance of critical structures are disclosed. Such surgical visualization systems can provide valuable information to clinicians during surgical procedures. Thus, for example, the clinician knows that the surgical visualization system is tracking, for example, critical structures accessible during anatomy (such as the ureter, specific nerves, and / or critical blood vessels), and can confidently maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide the clinician with instructions for a sufficiently long period to allow the clinician to pause and / or slow down the surgery and assess proximity to critical structures to prevent accidental damage. The surgical visualization system can provide the clinician with an ideal, optimized, and / or customizable amount of information to allow the clinician to confidently and / or rapidly maneuver through tissues while avoiding accidental damage to healthy tissues and / or critical structures, and thus minimizing the risk of injury caused by the surgery.

[0127] Figure 1This is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In some cases, the surgical visualization system 100 can be used intraoperatively to provide clinicians with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during surgical procedures. The surgical visualization system 100 is configured to identify critical structures intraoperatively and / or facilitate the avoidance of critical structures 101 by surgical instruments. For example, by identifying critical structures 101, clinicians can avoid manipulating surgical instruments around areas within a predetermined proximity of critical structures 101 during surgery. For example, clinicians can avoid dissecting veins, arteries, nerves, and / or blood vessels, such as those identified as critical structures 101, and / or avoid dissecting near these critical structures. In various cases, the critical structure 101 can be determined based on different patients and / or different surgical procedures.

[0128] The surgical visualization system 100 incorporates a distance sensor system 104 that integrates tissue identification and geometric surface mapping. Combined, these features of the surgical visualization system 100 can determine the orientation of a key structure 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the key structure 101. Furthermore, the surgical visualization system 100 includes an imaging system comprising, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various cases, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Views from the imaging device 120 can be provided to a clinician, and in various aspects of this disclosure, these views can be enhanced with additional information based on tissue identification, topographic mapping, and the distance sensor system 104. In such cases, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide clinicians with advanced data synthesis and integration information during surgery.

[0129] Imaging devices may include cameras or imaging sensors configured to detect, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). In various aspects of this disclosure, imaging systems may include, for example, imaging devices such as endoscopes. Additionally or alternatively, imaging systems may include, for example, imaging devices such as arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, colonoscopes, esophagogastric-duodenoscopes (gastroscopes), laryngoscopes, nasopharyngoscopes-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, or external endoscopes. In other cases, such as in open surgical applications, the imaging system may not include an observation endoscope.

[0130] In all respects of this disclosure, the tissue identification subsystem can be implemented using a spectral imaging system. The spectral imaging system may rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” published March 1, 2016, the entire contents of which are incorporated herein by reference.

[0131] In various aspects of this disclosure, the surface mapping subsystem can be implemented using an optical patterning system, as further described herein. The use of optical patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be used in the surgical visualization system described herein.

[0132] Structured light is the process of projecting a known pattern (typically a grid or horizontal stripes) onto a surface. U.S. Patent Application Publication 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published March 2, 2017, and U.S. Patent Application Publication 2017 / 0251900, entitled "DEPICTION SYSTEM," published September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. The full text of U.S. Patent Application Publication 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," and U.S. Patent Application Publication 2017 / 0251900, entitled "DEPICTION SYSTEM," published September 7, 2017, is incorporated herein by reference.

[0133] In various aspects of this disclosure, the distance determination system can be incorporated into a surface mapping system. For example, structured light can be used to generate a three-dimensional virtual model of a visible surface and determine various distances relative to the visible surface. Alternatively or additionally, the distance determination system can rely on time-of-flight measurements to determine one or more distances to tissue (or other structures) identified at a surgical site.

[0134] Figure 2 This is a schematic diagram of a control system 133 that can be used with a surgical visualization system 100. The control system 133 includes control circuitry 132 that communicates signalally with a memory 134. The memory 134 stores instructions executable by the control circuitry 132 to determine and / or identify critical structures (e.g., Figure 1 The key structure 101 in the system determines and / or calculates one or more distances and / or three-dimensional digital representations, and transmits certain information to one or more clinicians. For example, memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. Control system 133 also includes imaging system 142 having one or more cameras 144 (e.g., ...). Figure 1 The camera 144 may include one or more imaging devices 120, one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 may include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, three-dimensional lenses, etc.) that emit light in the various visible and invisible spectra. The display 146 may include one or more screens or monitors for displaying real, virtual, and / or virtual-enhanced images and / or information to one or more clinicians.

[0135] In all respects, the heart of the camera 144 is the image sensor 135. Generally speaking, a modern image sensor 135 is a solid-state electronic device containing up to millions of discrete photodetector sites (called pixels). Image sensor 135 technology falls into one of two categories: charge-connected devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) imagers, and recently, short-wave infrared (SWIR) has become an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) with bumps bonded to the CCD imaging substrate. Both CCD and CMOS image sensors 135 are sensitive to wavelengths from approximately 350 nm to 1050 nm, but this range is typically given as 400 nm to 1000 nm. Generally, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and therefore cannot distinguish colors. Therefore, two types of color CCD cameras exist: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to split incident light into a series of colors and employing interpolation algorithms to resolve the panchromatic image. Each color is then assigned to a different set of pixels. Three-chip color CCD cameras provide higher resolution by using prisms to direct each portion of the incident spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value, rather than using algorithms to determine color. Three-chip cameras offer extremely high resolution.

[0136] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source may be pulsed to emit wavelengths of light within the range of the spectral light source 150 and the range of light within the range of the structured light source 152. Alternatively, a single light source may be pulsed to provide wavelengths of light in the invisible spectrum (e.g., infrared light) and light in the visible spectrum. The spectral light source 150 may be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic unit 140 may identify critical structures via data received from the spectral light source 150 by the image sensor 135 portion of the camera 144. The surface mapping logic unit 136 may determine the surface profile of the visible tissue based on the reflected structured light. Using the time-of-flight measurement results, the distance determination logic unit 141 may determine one or more distances to the visible tissue and / or critical structure 101. One or more outputs from the surface mapping logic unit 136, the tissue identification logic unit 140, and the distance determination logic unit 141 may be provided to the imaging logic unit 138 and may be combined, mixed, and / or overlapped to be communicated to a clinician via the display 146 of the imaging system 142.

[0137] The instruction manual now briefly goes to Figures 2A to 2C This describes various aspects of the control circuitry 132 used to control various aspects of the surgical visualization system 100. (Go to...) Figure 2A This illustration shows a control circuit 400 configured to control various aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure. The control circuit 400 may be configured to implement the various processes described herein. The control circuit 400 may include a microcontroller including one or more processors 402 (e.g., microprocessor, microcontroller) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions to implement the various processes described herein. The processor 402 may be any of a variety of single-core or multi-core processors known in the art. The memory circuit 404 may include volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0138] Figure 2B A combinational logic circuit 410, configured to control various aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure, is shown. The combinational logic circuit 410 may be configured to implement the various processes described herein. The combinational logic circuit 410 may include a finite state machine including a combinational logic component 412 configured to receive data associated with a surgical instrument or tool at input 414, process the data through the combinational logic component 412, and provide an output 416.

[0139] Figure 2C A sequential logic circuit 420, configured to control various aspects of a surgical visualization system 100 according to at least one aspect of this disclosure, is shown. The sequential logic circuit 420 or combinational logic element 422 may be configured to implement the various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, combinational logic element 422, at least one memory circuit 424, and a clock 429. At least one memory circuit 424 may store the current state of the finite state machine. In some cases, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic element 422 is configured to receive data associated with a surgical device or system from input 426, process the data through the combinational logic element 422, and provide an output 428. In other aspects, the circuit may include a processor (e.g., Figure 2AThe various processes described herein are implemented by combining a processor 402 and a finite state machine. In other aspects, the finite state machine may include combinational logic circuits (e.g., combinational logic circuit 410, ...). Figure 2B The combination of ) and sequential logic circuit 420.

[0140] See you again Figure 1 In the surgical visualization system 100, the key structure 101 can be an anatomical structure of interest. For example, the key structure 101 can be an anatomical structure such as the ureter, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other cases, the key structure 101 can be, for example, an external structure in the anatomical field, such as a surgical device, surgical fastener, clamp, pin, probe, band, and / or plate. Exemplary key structures are further described herein and in co-filed U.S. patent applications (including, for example, U.S. Patent Application No. 16 / 128,192, filed September 11, 2018, entitled “VISUALIZATION OF SURGICAL DEVICES”), the entire contents of which are incorporated herein by reference.

[0141] In one aspect, the key structure 101 may be embedded within the tissue 103. In other words, the key structure 101 may be located below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the key structure 101 from view by the clinician. From the perspective of the imaging device 120, the key structure 101 is also obscured by the tissue 103. The tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other cases, the key structure 101 may be partially obscured, making it invisible.

[0142] Figure 1 Surgical device 102 is also depicted. Surgical device 102 includes an end effector having opposing jaws extending from the distal end of the axis of surgical device 102. Surgical device 102 can be any suitable surgical device, such as, for example, a dissecting instrument, suture device, gripper, applicator, and / or energy device (including monopolar probes, bipolar probes, ablation probes, and / or ultrasound end effectors). Alternatively or additionally, surgical device 102 may include, for example, another imaging or diagnostic modality, such as an ultrasound device. In one aspect of this disclosure, surgical visualization system 100 can be configured to enable the identification of one or more key structures 101 and the proximity of surgical device 102 to key structures 101.

[0143] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (such as a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens used together to simultaneously record two two-dimensional images, and thus generate a three-dimensional image of the surgical site, render the three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicating the morphology of visible tissue and the identification and orientation of hidden key structures, as further described herein. For example, the field of view of the imaging device 120 may overlap with a pattern of light (structured light) on the surface 105 of the tissue, such as… Figure 1 As shown.

[0144] In one aspect, the surgical visualization system 100 may be integrated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 include rigid structural members 116 and joints 118, which may include servo motor controls. The first robotic arm 112 is configured to manipulate a surgical device 102, and the second robotic arm 114 is configured to manipulate an imaging device 120. A robot control unit may be configured to issue control movements to the robotic arms 112 and 114, which may affect, for example, the surgical device 102 and the imaging device 120.

[0145] The surgical visualization system 100 also includes an emitter 106 configured to emit patterns of light, such as stripes, grid lines, and / or dots, to enable the determination of the topography or topography of surface 105. For example, a projection light array 130 can be used for three-dimensional scanning and registration on surface 105. The projection light array 130 can be emitted from the emitter 106 located, for example, on one of surgical devices 102 and / or robotic arms 112, 114 and / or imaging device 120. In one aspect, the projection light array 130 is used to determine the shape defined by the surface 105 of tissue 103 and / or the movement of the surface 105 during surgery. The imaging device 120 is configured to detect the projection light array 130 reflected from surface 105 to determine the topography of surface 105 and various distances relative to surface 105.

[0146] In one aspect, the imaging device 120 may further include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positioned by a robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, spectrometer, or vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable the identification of the type of anatomical and / or physical structures (such as the critical structure 101). Identification of the critical structure 101 may be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform transmitter 123 may include, for example, a photoacoustic imaging system. In other cases, the optical waveform transmitter 123 may be positioned on a surgical device separate from the imaging device 120.

[0147] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at a surgical site. In one aspect, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including a transmitter (such as transmitter 106) and a receiver 108 positionable on the surgical device 102. In other cases, the time-of-flight transmitter may be separate from the structured light transmitter. In a general aspect, the transmitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a mating sensor. The time-of-flight distance sensor system 104 can detect the “time of flight,” or the time taken for the laser emitted by transmitter 106 to bounce back to the sensor portion of receiver 108. The use of a very narrow light source in transmitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of tissue 103 directly in front of the distance sensor system 104. See still. Figure 1 d e It is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and d t This refers to the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. The distance sensor system 104 can be used to determine the transmitter-tissue distance d. e Device-tissue distance d tThe device-tissue distance d can be obtained from the known orientation of the transmitter 106 relative to the distal end of the surgical device 102 on its axis. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d is... t Based on the transmitter-to-organization distance d e Determined. In some cases, the axis of the surgical device 102 may include one or more articulated joints and may be capable of articulation relative to the transmitter 106 and the jaws. The articulated configuration may include, for example, a multi-joint vertebral structure. In some cases, a three-dimensional camera may be used to triangulate one or more distances to surface 105.

[0148] In various cases, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate surgical device rather than on surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which surgical device 102 extends to reach the surgical site. In other cases, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate robot-controlled arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In some cases, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance from the transmitter 106 on surgical device 102 to the surface 105 of tissue 103 using a line between the transmitter 106 on surgical device 102 and the imaging device 120. For example, the distance d may be determined based on the known orientation of the transmitter 106 (on surgical device 102) and the receiver 108 (on imaging device 120) of the time-of-flight distance sensor system 104. e Triangulation is performed. The three-dimensional orientation of receiver 108 can be known and / or registered with the robot coordinate plane during surgery.

[0149] In some cases, the orientation of the transmitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the orientation of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other cases, the surgical visualization system 100 can be used separately from the robotic system. In such cases, the distance sensor system 104 can operate independently of the robotic system.

[0150] In some cases, one or more of the robotic arms 112, 114 may be detached from the main robotic system used in the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and registered with a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or register the orientation of the robotic arms 112, 114 relative to a specific coordinate system. Similarly, the orientation of the surgical device 102 and the imaging device 120 may be registered with a specific coordinate system.

[0151] See still Figure 1 d w It is the camera-critical structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the critical structure 101, and d A This is the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time-of-flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d. w The use of spectral imaging combined with a time-of-flight sensor is further described in this paper. Additionally, see now... Figure 3 In various aspects of this disclosure, the depth d of the key structure 101 relative to the surface 105 of the tissue 103 A This can be determined by the following method: based on the distance d w and the known orientations of transmitter 106 on surgical device 102 and optical waveform transmitter 123 on imaging device 120 (and therefore the known distance d between them). x Triangulation is performed to determine the distance d. y (where d is the distance) e and d A sum).

[0152] Alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or waveform range) can be used to determine the camera-critical structure distance d. w Furthermore, the second waveform (or waveform range) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0153] Alternatively or alternatively, in some cases, distance d A It can be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance dA Spectral imaging can be used to determine this because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first manner or in a first amount, and collagen can reduce the detection signal in a different second manner or in a second amount.

[0154] See now Figure 4 The surgical visualization system 160 includes a surgical device 162 comprising an optical waveform transmitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform transmitter 123 may be configured to emit waveforms for determining a distance d from a common device (such as the surgical device 162). t and d w As further described herein. In such cases, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is... A This can be determined as follows:

[0155] d A =d w -d t .

[0156] As disclosed herein, various information regarding visible tissue, embedded key structures, and surgical devices can be determined using a combined approach that integrates an image sensor configured to detect spectral wavelengths and structured light arrays with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Furthermore, the image sensor can be configured to receive visible light and thus provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and to render a three-dimensional image of the surface tissue and underlying anatomical structures. In various cases, the imaging device 120 may include multiple image sensors.

[0157] Camera-critical structure distance d w Detection can also be performed using one or more alternative methods. In one aspect, key structures 201 can be illuminated using techniques such as fluorescence visualization (e.g., fluorescent indocyanine green (ICG)). Figures 6 to 8 As shown. Camera 220 may include two optical waveform sensors 222 and 224, which simultaneously capture left and right images of the key structure 201. Figure 7A and Figure 7B In such cases, camera 220 can depict the glow of the key structure 201 beneath the surface 205 of tissue 203, and at a distance of d. wThe distance can be determined from the known distance between sensors 222 and 224. In some cases, the distance can be determined more accurately by using more than one camera or by moving the camera between multiple positions. In some aspects, one camera may be controlled by a first robotic arm, and a second camera may be controlled by another robotic arm. In such robotic systems, a camera may be, for example, a follower camera on a follower arm. The follower arm and the camera on it may be programmed to track another camera and maintain, for example, a specific distance and / or lens angle.

[0158] In other aspects, the surgical visualization system 100 may employ two separate waveform receivers (i.e., camera / image sensor) to determine d w See now. Figure 9 If the critical structure 301 or its contents (e.g., blood vessels or vascular contents) can emit signals 302 using fluorescence fluoroscopy, the actual location can be triangulated based on two separate cameras 320a, 320b at a known location.

[0159] On the other hand, see now Figure 10A and Figure 10B The surgical visualization system can use a shake or move camera 440 to determine the distance d. w Camera 440 is robotically controlled, allowing its three-dimensional coordinates at different orientations to be known. In various situations, camera 440 can pivot at the cannula or patient interface. For example, if a critical structure 401 or its contents (e.g., a blood vessel or its contents) can emit signals, such as using fluorescence fluoroscopy, the actual position can be triangulated based on camera 440 rapidly moving between two or more known locations. Figure 10A In this process, camera 440 moves axially along axis A. More specifically, camera 440 translates a distance d1 along axis A closer to critical structure 401 to a position indicated as location 440', such as by moving in and out on a robotic arm. The distance to critical structure 401 can be calculated as camera 440 moves a distance d1 and the size of the view changes relative to critical structure 401. For example, an axial translation of 4.28 mm (distance d1) could correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Alternatively or additionally, camera 440 can rotate or sweep along arcs between different orientations. See now. Figure 10B Camera 440 moves axially along axis A and rotates about axis A by an angle θ3. The pivot point 442 for the rotation of camera 440 is located at the cannula / patient interface. Figure 10B In this process, camera 440 is translated and rotated to position 440". As camera 440 is moved and the view edges change with respect to key structure 401, the distance to key structure 401 can be calculated. Figure 10BIn this case, the distance d2 can be, for example, 9.01 mm, and the angle θ3 can be, for example, 0.9 degrees.

[0160] Figure 5 A surgical visualization system 500 is depicted, which is similar to surgical visualization system 100 in many respects. In various cases, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light of multiple wavelengths to obtain spectral images of, for example, hidden structures. In various cases, imaging device 520 may also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown intraoperatively for identifying and facilitating the avoidance of certain critical structures not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (uterus in this example).

[0161] The surgical visualization system 500 is configured to determine the emitter-tissue distance d from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to visualize based on the emitter-tissue distance d. e The device extends from the surgical device 502 to the surface 505 of the uterus 503 via a tissue distance d. t The surgical visualization system 500 is further configured to determine the tissue-ureteral distance d from the ureter 501a to the surface 505. A And the camera-ureter distance d from imaging device 520 to ureter 501a w As this article discusses... Figure 1 For example, the surgical visualization system 500 may utilize, for example, spectral imaging and time-of-flight sensors to determine the distance d. w In various situations, the surgical visualization system 500 can determine (e.g., triangulation) the tissue-ureter distance d based on other distance and / or surface mapping logic components described herein. A (or depth).

[0162] See now Figure 11The diagram depicts a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For instance, the control system 600 is a conversion system that integrates spectral feature tissue recognition and structured optical tissue localization to identify key structures, particularly when these structures are obscured by other tissues such as fat, connective tissue, blood, and / or other organs. Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue within an organ from healthy tissue.

[0163] The control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are utilized to detect and identify anatomical structures in a surgical field of view. The control system 600 includes conversion logic circuitry 648 to transform tissue data into information usable by the surgeon. For example, key structures within the anatomical structure can be identified using variable reflectivity based on the wavelength relative to the masking material. Furthermore, the control system 600 combines the identified spectral features and structured light data into an image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical use in a system with enhanced image overlay. Additional visual information can be used to employ the technology both intraoperatively and preoperatively. In various situations, the control system 600 is configured to provide warnings to clinicians upon approach to one or more key structures. Various algorithms can be employed to guide robotic automation and semi-automation methods based on surgical procedures and proximity to key structures.

[0164] Projected light arrays are used to determine tissue shape and movement intraoperatively. Alternatively, flash lidar can be used for surface mapping of tissue.

[0165] The control system 600 is configured to detect critical structures and provide image overlays of the critical structures, and to measure distances to the surface of visible tissue and to the embedded / buried critical structures. In other cases, the control system 600 may measure distances to the surface of visible tissue or detect critical structures and provide image overlays of the critical structures.

[0166] The control system 600 includes a spectral control circuit 602. For example, the spectral control circuit 602 may be a field-programmable gate array (FPGA) or, as described herein, a... Figures 2A to 2CAnother suitable circuit configuration is described. The spectral control circuit 602 includes a processor 604 to receive video input signals from a video input processor 606. For example, the processor 604 may be configured for hyperspectral processing and may utilize C / C++ code. For example, the video input processor 606 receives video input control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signals from the video input processor 606 and provide video output signals to a video output processor 608, which includes, for example, hyperspectral video output of interface control (metadata) data. The video output processor 608 provides the video output signals to an image overlay controller 610.

[0167] A video input processor 606 is coupled to a camera 612 on the patient side via a patient isolation circuit 614. As previously described, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers to isolate the patient from other circuitry in the system. The camera 612 receives intraoperative images via optics 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include, for example, [other types described herein]. Figure 2 Any image sensor technology described herein. In one aspect, camera 612 outputs an image with a 14-bit / pixel signal. It should be understood that higher or lower pixel resolutions may be used without departing from the scope of this disclosure. An isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which utilizes hardware register 618 and a Nios2 coprocessor 620 to process the camera output signal 613. The color RGB fused output signal is provided to a video input processor 606 and a laser pulse control circuit 622.

[0168] Laser pulse control circuit 622 controls laser engine 624. Laser engine 624 outputs light of multiple wavelengths (λ1, λ2, λ3...λn), including near-infrared (NIR). Laser engine 624 can operate in multiple modes. In one aspect, laser engine 624 can operate in, for example, two modes. In the first mode (e.g., normal operation mode), laser engine 624 outputs an illumination signal. In the second mode (e.g., identification mode), laser engine 624 outputs RGBG and NIR light. In various cases, laser engine 624 can operate in polarization mode.

[0169] Light output 626 from laser engine 624 illuminates the target anatomical structure in surgical site 627 during surgery. Laser pulse control circuitry 622 also controls laser pulse controller 628 for laser patterning projector 630, which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) onto the surgical tissue or organ at surgical site 627. Camera 612 receives the patterned light and reflected light output through camera optics 632. Image sensor 634 converts the received light into digital signals.

[0170] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and the video input module 636 for reading the laser pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser pattern 631 and outputs a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0171] The first video output signal 640 and the second video output signal 642 include data representing the orientation of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. Combined with data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine the distance d to the buried critical structure. A ( Figure 1 (For example, via triangulation algorithm 644), and distance d A The video output processor 646 can provide the output to the image overlay controller 610. The aforementioned conversion logic components may include the conversion logic circuit 648, the intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0172] In various situations, preoperative data 650 from CT or MRI scans can be used to register or match certain three-dimensional deformable tissues. This preoperative data 650 can be provided to an integration module 643 and ultimately to an image overlay controller 610, allowing this information to be overlaid with a view from a camera 612 and provided to a video monitor 652. The registration of preoperative data is further described herein and in the foregoing concurrently filed U.S. patent applications (including, for example, U.S. Patent Application No. 16 / 128,195, filed September 11, 2018, entitled “INTEGRATION OF IMAGING DATA”), the entire contents of which are incorporated herein by reference.

[0173] Video monitor 652 can output an integrated / enhanced view from image overlay controller 610. Clinicians can select and / or switch between different views on one or more monitors. On the first monitor 652a, a clinician can switch between (A) a view depicting a 3D rendering of visible tissue and (B) an enhanced view depicting one or more hidden critical structures on the 3D rendering of visible tissue. On the second monitor 652b, a clinician can, for example, switch distance measurements to the surface of one or more hidden critical structures and / or visible tissue.

[0174] The control system 600 and / or its various control circuits can be integrated into the various surgical visualization systems disclosed herein.

[0175] Figure 12 A structured (or patterned) light system 700 according to at least one aspect of this disclosure is illustrated. As described herein, structured light in the form of stripes or lines can be projected, for example, from a light source and / or a projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contour of the surface 705. In various aspects, it may be similar to imaging device 120 ( Figure 1 The camera 720 can be configured, for example, to detect the pattern of light projected onto the surface 705. The way the projected pattern deforms upon impact with the surface 705 allows the vision system to calculate depth and surface information of the target's anatomy.

[0176] In some cases, invisible (or imperceptible) structured light can be used, where it can be employed without interfering with other computer vision tasks where the projected pattern might become confuse. For example, alternating infrared light or extremely fast visible light frame rates between two completely opposite patterns can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0177] As described above, the various surgical visualization systems described herein can be used to visualize various types of tissues and / or anatomical structures, including those that are obscured by EMR in the visible portion of the spectrum and thus cannot be visualized. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissues based on different combinations of constituent materials. Specifically, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the absorption coefficients of the tissue at various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on specific combinations of constituent materials. For illustration, Figure 13AThis is graph 2300 depicting how the absorption coefficients of various biomaterials vary across the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient of the biomaterial (e.g., in cm⁻¹). -1 The horizontal axis 2304 represents the EMR wavelength (e.g., in μm). Graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of proteins at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxyhemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxyhemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials; therefore, tissue types visualized by a surgical visualization system can be identified and distinguished based on specific combinations of detected constituent materials. Thus, a spectral imaging system can be configured to emit multiple different wavelengths of EMR, determine the constituent materials of the tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on specific detection combinations of constituent materials.

[0178] Figure 13B This demonstrates the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures. Figure 13B In this imaging system, a spectral emitter 2320 (e.g., a spectral light source 150) is used to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissue and / or structures at the surgical site 2325 can be visualized by an image sensor 135. Figure 2 The imaging system 142 receives data to visualize tissue and / or structures; these tissues and / or structures may be visible (e.g., located on the surface of surgical site 2325) or obscured (e.g., located below other tissues and / or structures at surgical site 2325). In this example, the imaging system 142 ( Figure 2 The imaging system 142 can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissues whose spectral characteristics do not conform to known or expected spectral characteristics) based on spectral features characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials of each of the different tissue / structure types. The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system 142, such as imaging system display 146. Figure 2 ), Main display 2119 ( Figure 18 ), non-sterile display 2109 ( Figure 18 ), Hub Display 2215 ( Figure 19), Device / Instrument Display 2237 ( Figure 19 )wait.

[0179] Furthermore, the imaging system 142 can be configured to customize or update the visualization of the displayed surgical site based on the identified tissue and / or structure type. For example, the imaging system 142 can display on a display screen (e.g., monitor 146) an edge 2330a associated with the tumor 2332 being visualized. The edge 2330a can indicate the area or amount of tissue that should be removed to ensure complete removal of the tumor 2332. Control system 133 ( Figure 2 The system 142 can be configured to control or update the size of edge 2330a based on tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified multiple anomalies 2338 within the field of view (FOV). Therefore, the control system 133 can adjust the displayed edge 2330a to a first updated edge 2330b, which has a sufficient size to cover the anomalies 2338. Furthermore, the imaging system 142 has also identified an artery 2334 (as shown by the highlighted area 2336 of artery 2334) that partially overlaps with the initially displayed edge 2330a. Therefore, the control system 133 can adjust the displayed edge 2330a to a second updated edge 2330c, which has a sufficient size to cover the relevant portion of artery 2334.

[0180] In addition to the above, regarding Figure 13A and 13B In addition to or in lieu of the described absorption properties, tissues and / or structures can also be imaged or characterized on EMR wavelength spectra based on their reflectance properties. For example, Figures 13C-13E Various graphs showing the reflectance of different types of tissues or structures at different EMR wavelengths are presented. Figure 13C This is a graphical representation of the ureteral features relative to the obscuring material 1050. Figure 13D This is a graphical representation of the illustrative arterial features relative to the obscuration 1052. Figure 13E It is a graphical representation of an exemplary neural feature relative to an obscuring object 1054. Figures 13C-13E The curves in the graphs represent the reflectance of specific structures (ureters, arteries, and nerves) relative to fat, lung tissue, and blood at corresponding wavelengths as a function of wavelength (nm). These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may have corresponding detectable reflective features that would allow for the identification and visualization of tissues and / or structures.

[0181] In various scenarios, selected wavelengths for spectral imaging can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site (i.e., “selective spectral” imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized, enabling information to be acquired in real-time or near real-time and utilized during surgery. In various scenarios, the wavelength can be selected by the clinician or by control circuitry based on clinician input. In some cases, the wavelength can be selected based on, for example, machine learning and / or large datasets accessible to the control circuitry via the cloud.

[0182] The aforementioned application of spectral imaging to tissue can be used during surgery to measure the distance between a waveform transmitter and critical structures obscured by tissue. In one aspect of this disclosure, see now. Figure 14 and Figure 15 The diagram illustrates a time-of-flight sensor system 1104 utilizing waveforms 1124 and 1125. In some cases, the time-of-flight sensor system 1104 can be integrated into a surgical visualization system 100. Figure 1 The time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. A transmitted wave 1124 extends from the transmitter 1106 to a critical structure 1101, and a received wave 1125 is reflected back from the critical structure 1101 by the receiver 1108. The surgical device 1102 is positioned through a cannula 1110 extending into a lumen 1107 in the patient.

[0183] Waveforms 1124 and 1125 are configured to penetrate the obscured tissue 1103. For example, the wavelengths of waveforms 1124 and 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal may be emitted from transmitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, waveforms 1124 and 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, transmitter 1106 is configured to provide binary signal on and off, such as Figure 15 As shown, for example, this binary signal can be measured by receiver 1108.

[0184] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d. Figure 14 ). Figure 14The flight time timing diagram 1130 of transmitter 1106 and receiver 1108 is in Figure 15 As shown in the figure. The delay is a function of the distance d, and the distance d is given by the following equation:

[0185]

[0186] in:

[0187] c = speed of light;

[0188] t = length of the pulse;

[0189] q11 = the charge accumulated during light emission; and

[0190] q2 = The charge accumulated when no light is emitted.

[0191] As provided in this article, the flight times of waveforms 1124 and 1125 correspond to Figure 14 The distance d in the distance. In various cases, the additional transmitter / receiver and / or the pulse signal from transmitter 1106 can be configured to transmit a non-penetrating signal. The non-penetrating tissue can be configured to determine the distance from the transmitter to the surface 1105 of the shielding tissue 1103. In various cases, the depth of the critical structure 1101 can be determined by the following formula:

[0192] d A =d w -d t .

[0193] in:

[0194] d A = Depth of critical structure 1101;

[0195] d w = Distance from transmitter 1106 to critical structure 1101 ( Figure 14 d) in; and

[0196] d t = The distance from the transmitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the shielding tissue 1103.

[0197] In one aspect of this disclosure, see now. Figure 16 The diagram illustrates a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, and 1225c. In some cases, the time-of-flight sensor system 1204 can be integrated into a surgical visualization system 100. Figure 1The time-of-flight sensor system 1204 includes a waveform transmitter 1206 and a waveform receiver 1208. The waveform transmitter 1206 is positioned on a first surgical device 1202a, and the waveform receiver 1208 is positioned on a second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their respective cannulas 1210a and 1210b, which extend into the patient's lumen 1207. Transmitted waves 1224a, 1224b, and 1224c extend from the transmitter 1206 toward the surgical site, and received waves 1225a, 1225b, and 1225c are reflected back to the receiver 1208 from various structures and / or surfaces at the surgical site.

[0198] Different emitted waves 1224a, 1224b, and 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, and 1224c can be in the visible, NIR, or SWIR wavelength spectrum. For example, visible light can be reflected from the surface 1205 of tissue 1203, and NIR and / or SWIR waveforms can be configured to penetrate the surface 1205 of tissue 1203. In various aspects, as described herein, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from emitter 1206. In various cases, waves 1224b and 1224c can be selected based on the spectral characteristics of key structures 1201a and 1201b to target key structures 1201a and 1201b within tissue 1203, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, which are incorporated herein by reference.

[0199] The emitted waves 1224a, 1224b, and 1224c can be reflected from the target material (i.e., surface 1205, the first key structure 1201a, and the second structure 1201b, respectively). The received waveforms 1225a, 1225b, and 1225c can be reflected due to... Figure 16 The distance d shown 1a d 2a d 3a d 1b d 2b d 2c And thus delayed.

[0200] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 can be independently positioned (e.g., positioned on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d can be calculated based on the known orientations of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when surgical devices 1202a and 1202b are controlled by a robot, these orientations can be known. Knowledge of the orientations of the transmitter 1206 and receiver 1208, the timing of the photon stream targeting a specific tissue, and information about that particular response received by receiver 1208 allows for the determination of distance d. 1a d 2a d 3a d 1b d 2b d 2c In one aspect, the distance to the shielded critical structures 1201a and 1201b can be triangulated using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0201] See still Figure 16 In various situations, in the view provided to the clinician, receiver 1208 can be rotated such that the centroid of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such orientation can rapidly transmit one or more relevant distances and / or perspectives regarding key structures. For example, as... Figure 16 As shown, the surgical site is displayed from a viewpoint perpendicular to the viewing plane (i.e., with blood vessels oriented in / outside the page) of the key structure 1201a. In various cases, this orientation may be the default setting; however, the view can be rotated or otherwise adjusted by the clinician. In some situations, the clinician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0202] In various cases, receiver 1208 may be mounted on a cannula or endotracheal tube (such as cannula 1210b), through which surgical device 1202b is positioned. In other cases, receiver 1208 may be mounted on a separate robotic arm whose three-dimensional orientation is known. In various cases, receiver 1208 may be mounted on a movable arm separate from the robot controlling surgical device 1202a, or may be mounted on an operating room (OR) table that can be registered with the robot's coordinate plane during surgery. In such cases, the orientation of transmitter 1206 and receiver 1208 may be able to be registered with the same coordinate plane, allowing for distance triangulation based on the output from time-of-flight sensor system 1204.

[0203] The combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS) (referred to as TOF-NIRS, which is capable of measuring time-resolved characteristic maps of NIR light with nanosecond resolution) can be seen in the article entitled “TIME-OF-FLIGHT NEAR-INFRAREDSPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY INGRAPEFRUIT” (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228) (the full text of which is incorporated herein by reference) and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0204] In various scenarios, time-of-flight spectral waveforms are configured to determine the depth of critical structures and / or the proximity of surgical devices to these structures. Furthermore, the various surgical visualization systems disclosed herein include surface mapping logic components configured to create a 3D rendering of the surface of visible tissue. In such cases, clinicians can know the proximity (or lack thereof) of surgical devices to critical structures even when visible tissue obscures them. In one scenario, the topography of the surgical site is provided on a monitor by the surface mapping logic component. If a critical structure is close to the surface of the tissue, spectral imaging can convey the orientation of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on known limitations of spectral imaging systems, the system is configured to convey that a critical structure is outside the range even when the spectral imaging system cannot detect it at all. Therefore, clinicians can continue to move surgical devices and / or manipulate tissue. When a critical structure moves into the range of the spectral imaging system, the system can identify the structure and thus convey that the structure is within range. In such cases, an alert can be provided when the structure is initially identified and / or when the structure is further moved within a predefined proximity region. In these situations, even if the spectral imaging system fails to identify a critical structure using known boundaries / ranges, it can still provide proximity information (i.e., lack of proximity) to clinicians.

[0205] The various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and to alert clinicians before accidental dissection and / or transection damage to critical structures. In various aspects, the surgical visualization systems are configured to identify one or more critical structures, such as the ureter, intestine, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve, and their branches), blood vessels (including the pulmonary artery and lobar artery and pulmonary vein and lobar vein, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including the middle colic artery, right colic artery, and ileal artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal (lower abdomen) ileal vessels, short gastric arteries, uterine arteries, median sacral vessels, and lymph nodes. Furthermore, the surgical visualization systems are configured to indicate the proximity of surgical devices to critical structures and / or alert clinicians when surgical devices are close to critical structures.

[0206] Various aspects of this disclosure provide for the identification of critical structures (e.g., the ureter, nerves, and / or blood vessels) and monitoring of instrument proximity during surgery. For example, the various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, enabling the visualization of critical structures, for example, below the surface of tissue (e.g., 1.0 cm to 1.5 cm below the surface of tissue). In other cases, the surgical visualization system may identify structures less than 1.0 cm or greater than 1.5 cm below the surface of tissue. For example, a surgical visualization system that can identify structures, for example, within 0.2 mm of the surface, can be valuable if the structure would otherwise not be visible due to depth. In various aspects, the surgical visualization system may, for example, utilize a virtual depiction of the critical structure as a visible white light image superimposed on the surface of visible tissue to enhance the clinician's view. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal end of a surgical instrument and may provide proximity alerts, for example, when the distal end of the surgical instrument moves within a specific range of a critical structure (e.g., within 1.0 cm of the critical structure).

[0207] The various surgical visualization systems disclosed herein can identify when anatomy is too close to critical structures. Anatomy may be “too close” to a critical structure based on temperature (i.e., too hot near a critical structure where there is a risk of damage / heating / melting) and / or tension (i.e., too much tension near a critical structure where there is a risk of damage / tearing / traction). For example, such surgical visualization systems can be beneficial for anatomy around blood vessels when skeletalizing them before ligation. In various cases, a thermal imaging camera can be used to read the heat at the surgical site and provide warnings to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the temperature of the tool is above a predefined threshold (e.g., 120℉), a warning can be provided to the clinician at a first distance (e.g., 10 mm), and if the temperature of the tool is less than or equal to the predefined threshold, a warning can be provided at a second distance (e.g., 5 mm). The predefined thresholds and / or warning distances can be default settings and / or programmable by the clinician. Alternatively or concurrently, proximity warnings may be associated with thermal measurements taken by the tool itself, such as thermocouples that measure heat in the distal jaws of a monopolar or bipolar dissecter or vascular occluder.

[0208] The various surgical visualization systems disclosed herein provide sufficient sensitivity regarding critical structures and specificities, enabling clinicians to confidently perform rapid yet safe dissections based on standards of care and / or device safety data. The systems can operate in real-time during surgery with minimal or no risk of ionizing radiation to the patient or clinician in all cases. Conversely, during fluoroscopy, patients and clinicians may be exposed to ionizing radiation via, for example, X-ray beams used for real-time observation of anatomical structures.

[0209] When the path of a surgical device is controlled by a robot, the various surgical visualization systems disclosed herein can be configured to detect and identify, for example, one or more key structures of a desired type in the forward path of the surgical device. Alternatively, the surgical visualization system can be configured to detect and identify, for example, one or more key structures of a certain type in the region surrounding the surgical device and / or in multiple planes / dimensions.

[0210] The various surgical visualization systems disclosed herein are easy to operate and / or interpret. Furthermore, these systems can be combined with "overwrite" features that allow clinicians to override default settings and / or operations. For example, clinicians may selectively disable warnings from the surgical visualization system and / or move closer to the critical structure than suggested by the system when the risk to a critical structure is less than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue may be greater than the risk of damaging the critical structure).

[0211] The various surgical visualization systems disclosed herein can be integrated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, the specific implementation of a surgical visualization system may not change how surgery is performed. Furthermore, surgical visualization systems may be more economical compared to the cost of accidental transection. Data suggests that reduced accidental damage to critical structures can drive incremental compensation.

[0212] The various surgical visualization systems disclosed in this article can operate in real-time or near real-time and far in advance, enabling clinicians to anticipate critical structures. For example, surgical visualization systems can provide sufficient time to "slow down, assess, and avoid" in order to maximize the efficiency of surgical procedures.

[0213] The various surgical visualization systems disclosed herein may not require the injection of contrast agents or dyes into the tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other cases, contrast agents can be injected into the appropriate tissue layers more easily than with other visualization systems. For instance, the time between contrast agent injection and visualization of key structures may be less than two hours.

[0214] The various surgical visualization systems disclosed herein can be correlated with clinical data and / or device data. For example, the data can provide information about how close a power-enabled surgical device (or other potentially damaging device) should be to the boundary of tissue the surgeon does not want to damage. Any data modules that interact with the surgical visualization systems disclosed herein can be provided integrally with or separately from the robot, enabling their use in conjunction with stand-alone surgical devices in, for example, open or laparoscopic surgeries. In various cases, the surgical visualization systems can be compatible with robotic surgical systems. For example, visualized images / information can be displayed on the robot's control console.

[0215] In various situations, clinicians may not know the location of critical structures relative to surgical instruments. For example, when a critical structure is embedded in tissue, the clinician may be unable to determine its location. In some cases, clinicians may want to keep surgical devices outside the azimuth range surrounding the critical structure and / or away from visible tissue covering or concealing it. When the location of the concealed critical structure is unknown, clinicians may risk moving too close to the critical structure, potentially causing unintentional trauma and / or excessive energy, heat, and / or tension on the anatomy and / or vicinity of the critical structure. Alternatively, clinicians may keep too far from the suspected location of the critical structure and risk influencing the tissue in a less than ideal position to attempt to avoid the critical structure.

[0216] This invention provides a surgical visualization system that presents surgical device tracking relative to one or more key structures. For example, the surgical visualization system can track the proximity of a surgical device to a key structure. Such tracking can occur intraoperatively, in real-time, and / or near real-time. In various cases, the tracking data can be provided to the clinician via a display screen (e.g., a monitor) of an imaging system.

[0217] In one aspect of this disclosure, a surgical visualization system includes: a surgical device including an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect an embedded structure and the structured light pattern on the visible surface; and control circuitry in signal communication with the camera and the imaging system, wherein the control circuitry is configured to determine a distance from the surgical device to the embedded structure and to provide a signal indicative of that distance to the imaging system. For example, the distance can be determined by calculating the distance from the camera to a critical structure illuminated using fluorescence fluoroscopy and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses (e.g., a left lens and a right lens) of the camera. For example, the distance from the surgical device to the critical structure can be triangulated based on the known orientation of the surgical device and the camera. Alternative devices for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue superimposed / covering the embedded critical structure. For example, surgical visualization systems can identify and enhance the view of hidden critical structures by drawing a schematic diagram (such as a line on the surface of visible tissue) over visible structures. Surgical visualization systems can also determine the distance to the enhancement line on the visible tissue.

[0218] By providing clinicians with up-to-date information on the proximity of surgical devices to concealed critical structures and / or visible structures, as disclosed in this article through various surgical visualization systems, clinicians can make more informed decisions regarding the placement of surgical devices relative to concealed critical structures. For example, clinicians can view the distance between the surgical device and the critical structure in real time / intraoperatively, and in some cases, the imaging system can provide alerts and / or warnings when the surgical device moves into a predefined proximity and / or area of ​​the critical structure. In some cases, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly zone" near the critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more). In such cases, clinicians can maintain momentum throughout the surgical procedure without needing to monitor the suspected location of the critical structure and the proximity of the surgical device to it. Therefore, some surgical procedures can be performed more quickly with fewer pauses / interruptions and / or with improved accuracy and / or certainty. In one respect, surgical visualization systems can be used to detect tissue variability, such as the variability of tissues within organs, to distinguish between tumor / cancer / unhealthy tissue and healthy tissue. Such surgical visualization systems can maximize the removal of unhealthy tissue while minimizing the removal of healthy tissue.

[0219] Surgical hub system

[0220] The various visualization or imaging systems described in this article can be incorporated into surgical hub systems, such as by combining... Figures 17-19 It is shown and described in further detail below.

[0221] See Figure 17 The computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. In one example, as... Figure 17 As shown, the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and handheld intelligent surgical instruments 2112, which are configured to communicate with each other and / or with a hub 2106. In some aspects, the surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to one.

[0222] Figure 18 An example of a surgical system 2102 for performing surgery on a patient lying supine on an operating table 2114 in an operating room 2116 is shown. A robotic system 2110 is used as part of the surgical system 2102 during the surgery. The robotic system 2110 includes a surgeon's console 2118, a patient-side trolley 2120 (surgical robot), and a surgical robot hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side trolley 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision within the patient's body. Images of the surgical site can be obtained via a medical imaging device 2124, which can be manipulated by the patient-side trolley 2120 to orient the imaging device 2124. The robot hub 2122 can be used to process the images of the surgical site for subsequent display to the surgeon via the surgeon's console 2118.

[0223] Other types of robotic systems can be readily adapted for use with surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0224] Various examples of cloud-based analytics performed by Cloud 2104 and applicable to this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0225] In various respects, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0226] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate multiple portions of the surgical site. One or more image sensors may receive light reflected or refracted from the surgical site, including light reflected or refracted from tissue and / or surgical instruments.

[0227] One or more illumination sources can be configured to radiate electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or emission spectrum) is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by it) and can be called visible light or simple light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0228] The invisible spectrum (i.e., the non-luminescent spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0229] In various respects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, colonoscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0230] In one aspect, imaging devices employ multispectral monitoring to discern morphology and underlying structure. A multispectral image is an image that captures image data across a specific wavelength range of the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated herein by reference. After completing a surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site.

[0231] It goes without saying that rigorous sterilization of the operating room and surgical equipment is required during any surgical procedure. The stringent hygienic and sterilization conditions required in the “surgical room” (i.e., operating room or treatment room) necessitate the highest possible sterility of all medical devices and equipment. Part of this sterilization process requires the sterilization of any material that comes into contact with the patient or penetrates the sterile area, including the imaging device 2124 and its attachments and components. It should be understood that a sterile area can be considered a designated area deemed free of microorganisms, such as within a tray or sterile towel, or can be considered the area around the patient prepared for surgical procedures. A sterile area may include properly dressed scrubbed team members, as well as all equipment and fixtures within that area.

[0232] In various aspects, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays, strategically arranged relative to a sterile area, such as... Figure 18 As shown in the figure. In one aspect, visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of visualization system 2108 are described in various U.S. patent applications, which are incorporated herein by reference.

[0233] like Figure 18 As shown, the main display 2119 is positioned within the sterile area for visibility to the operator at the operating table 2114. Furthermore, a visualization tower 21121 is positioned outside the sterile area. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 positioned opposite each other. A visualization system 2108, guided by a hub 2106, is configured to utilize displays 2107, 2109, and 2119 to coordinate information flow to operators both inside and outside the sterile area. For example, the hub 2106 allows the visualization system 2108 to display snapshots of the surgical site recorded by the imaging device 2124 on the non-sterile displays 2107 or 2109 while maintaining a real-time feed of the surgical site on the main display 2119. The snapshots on the non-sterile displays 2107 or 2109 may allow a non-sterile operator to perform diagnostic steps related to the surgical procedure, for example.

[0234] In one aspect, hub 2106 is further configured to route diagnostic inputs or feedback entered by a non-sterile operator at visualization tower 21121 to a main display 2119 within a sterile area, which can be viewed by a sterile operator at a workbench. In one example, the input may be a modified form of a snapshot displayed on non-sterile displays 2107 or 2109, which can be routed to the main display 2119 via hub 2106.

[0235] See Figure 18 Surgical instrument 2112 is used in surgical procedures as part of surgical system 2102. Hub 2106 is further configured to coordinate information flow to the display of surgical instrument 2112, as described in various U.S. patent applications, which are incorporated herein by reference. Diagnostic input or feedback entered by a non-sterile operator at visualization tower 21121 can be routed by hub 2106 to surgical instrument display 2115 within a sterile area, where the operator of surgical instrument 2112 can observe the input or feedback. Exemplary surgical instruments suitable for surgical system 2102 are described in various U.S. patent applications, which are incorporated herein by reference.

[0236] Figure 19 An interactive surgical system 2200 implemented by a computer is illustrated. The interactive surgical system 2200 implemented by a computer is similar in many respects to the interactive surgical system 2100 implemented by a computer. The surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204, which may include a remote server 2213. In one aspect, the interactive surgical system 2200 implemented by a computer includes a surgical hub 2236 that connects to multiple operating room devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively connecting the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. Figure 19 As illustrated in the example, the surgical hub 2236 is connected to an imaging module 2238 (which is connected to an endoscope 2239), a generator module 2240 connected to an energy device 2421, a fume extractor module 2226, a suction / rinsing module 2228, a communication module 2230, a processor module 2232, a storage array 2234, an intelligent device / instrument 2235 optionally connected to a display 2237, and a non-contact sensor module 2242. Operating room equipment is connected to cloud computing resources and data storage via the surgical hub 2236. A robotic hub 2222 can also be connected to the surgical hub 2236 and cloud computing resources. Devices / instruments 2235, visualization systems 2209, etc., can be connected to the surgical hub 2236 via wired or wireless communication standards or protocols, as described herein. The surgical hub 2236 can be coupled to the hub display 2215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display, and / or other visualization system 208. The hub display can also combine and overlay images to display data received from devices connected to the modular control tower.

[0237] Situational awareness

[0238] The various visualization systems or aspects thereof described herein can be used as part of a situational awareness system, which can be comprised of surgical hubs 2106, 2236 ( Figures 17-19 The situational awareness system can use this situational data to implement or perform surgical procedures. Specifically, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and movement), tissues, structures, users, and other elements located in the surgical field or operating room can provide contextual data. The situational awareness system can then use this contextual data to infer the type of surgical procedure being performed or its steps, the type of tissue and / or structure the surgeon is manipulating, etc. The situational awareness system can then use this contextual data to provide alerts to the user, suggest subsequent steps or actions, prepare surgical devices for use (e.g., activate an electrosurgical generator as expected in a subsequent step of the surgical procedure using an electrosurgical instrument), intelligently control surgical instruments (e.g., customize surgical instrument operating parameters based on each patient's specific health condition), etc.

[0239] While a “smart” device that includes control algorithms responding to sensed data can be an improvement over a “dumb” device that operates without considering sensed data, some sensed data can be incomplete or uncertain when considered in isolation—that is, without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context (e.g., knowing the type of tissue being operated on or the type of surgery being performed), the control algorithm may incorrectly or suboptimally control the modular device given specific, context-free sensed data. Modular devices can include any surgical device that can be controlled by a situational awareness system, such as visualization system devices (e.g., cameras or displays), surgical instruments (e.g., ultrasound surgical instruments, electrosurgical instruments, or surgical sutures), and other surgical devices (e.g., fumigators). For example, the optimal approach for a control algorithm to control surgical instruments in response to specific sensed parameters can vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and therefore respond differently to actions taken by surgical instruments. Therefore, it may be expected that surgical instruments will act differently even when sensing the same measurement for a specific parameter. As a specific example, the optimal way to control surgical suturing and cutting instruments in response to unexpectedly high forces sensed by the instrument for closing its end effector will vary depending on whether the tissue type is prone to tearing or resistant to tearing. For easily tearable tissues (such as lung tissue), the instrument's control algorithm will optimally decrease the motor speed gradually in response to unexpectedly high forces for closure, thereby avoiding tissue tearing. For resistant tissues (such as stomach tissue), the instrument's control algorithm will optimally increase the motor speed gradually in response to unexpectedly high forces for closure, thereby ensuring that the end effector is properly clamped onto the tissue. In cases where it is unknown whether lung or stomach tissue has been clamped, the control algorithm may make a suboptimal decision.

[0240] One solution utilizes a surgical hub comprising a system configured to derive information about a surgical procedure being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about a surgical procedure from the received data, and then control modular devices paired with the surgical hub based on the inferred context of the surgical procedure. Figure 20A diagram of a situational awareness surgical system 2400 according to at least one aspect of this disclosure is shown. In some examples, the data source 2426 includes, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database containing patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0241] The surgical hub 2404 (which may be similar to hub 106 in many respects) may be configured to derive surgical context information from the data, for example, based on a specific combination of received data or a specific order in which data is received from data source 2426. The context information inferred from the received data may include, for example, the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity of the object of the procedure. This ability of the surgical hub 2404 to derive or infer surgical context information from the received data may be referred to as “situational awareness.” In one example, the surgical hub 2404 may be incorporated into a situational awareness system, which is the hardware and / or programming associated with the surgical hub 2404 for deriving surgical context information from the received data.

[0242] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data source 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to associate various inputs (e.g., data from the database 2422, patient monitoring device 2424, and / or modular device 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another example, the situational awareness system may include a lookup table that stores pre-represented contextual information about the surgical procedure associated with one or more inputs (or ranges of inputs) corresponding to the contextual information. In response to a query using one or more inputs, the lookup table can return the corresponding contextual information used by the situational awareness system to control the modular device 2402. In one example, the contextual information received by the situational awareness system of the surgical hub 2404 is associated with a specific control adjustment or a set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes additional machine learning systems, lookup tables, or other such systems that generate or retrieve one or more control adjustments for one or more modular devices 2402 when provided with contextual information as input.

[0243] The surgical hub 2404, incorporating a situational awareness system, provides numerous benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or data utilization during surgical procedures. Returning to the previous example, the situational awareness surgical hub 2404 can determine the type of tissue being operated on; therefore, when an unexpectedly high force is detected on the end effector of the surgical instrument used for closure, the situational awareness surgical hub 2404 can correctly adjust the motor speed of the surgical instrument according to the tissue type by gradually increasing or decreasing it.

[0244] As another example, the type of tissue being operated on can influence the adjustment of the compression rate and load threshold of surgical suture and cutting instruments for specific tissue gap measurements. The situation-aware surgical hub 2404 can infer whether the surgery being performed is thoracic or abdominal, allowing it to determine whether the tissue held by the end effector of the surgical suture and cutting instruments is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical suture and cutting instruments according to the tissue type.

[0245] As another example, the type of body cavity operated on during a blow-through procedure can affect the function of the smoke extractor. The situation-aware surgical hub 2404 can determine whether the surgical site is under pressure (by determining that a surgical procedure is being performed using blow-through) and determine the type of surgery. Since a type of surgery is typically performed within a specific body cavity, the surgical hub 2404 can then appropriately control the motor speed of the smoke extractor for the body cavity in which the procedure is being performed. Therefore, the situation-aware surgical hub 2404 can provide consistent smoke extraction for both thoracic and abdominal surgeries.

[0246] As yet another example, the type of procedure being performed can influence the optimal energy level for operating ultrasound surgical instruments or radiofrequency (RF) electrosurgical instruments. For instance, arthroscopic procedures require higher energy levels because the end effectors of the ultrasound surgical instruments or RF electrosurgical instruments are immersed in fluid. A situational-aware surgical hub 2404 can determine whether a surgical procedure is arthroscopic. The surgical hub 2404 can then adjust the RF power level or ultrasound amplitude (i.e., the “energy level”) of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can influence the optimal energy level for operating ultrasound surgical instruments or RF electrosurgical instruments. The situational-aware surgical hub 2404 can determine the type of surgical procedure being performed and then tailor the energy levels of the ultrasound surgical instruments or RF electrosurgical instruments separately based on the expected tissue profile of that surgical procedure. Furthermore, the situational-aware surgical hub 2404 can be configured to adjust the energy levels of the ultrasound surgical instruments or RF electrosurgical instruments throughout the entire surgical procedure, rather than just on a per-procedure basis. The situational awareness surgical hub 2404 can determine the steps of a surgical procedure being performed or to be performed subsequently, and then update the control algorithms for the generator and / or ultrasound surgical instruments or RF electrosurgical instruments to set the energy level to a value suitable for the expected tissue type based on the surgical step.

[0247] As another example, data can be extracted from additional data source 2426 to improve the conclusions drawn by surgical hub 2404 from one data source 2426. The situational-aware surgical hub 2404 can augment the data received from modular device 2402 with background information about the surgical procedure already constructed from other data sources 2426. For example, the situational-aware surgical hub 2404 can be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) based on video or image data received from a medical imaging device. However, in some cases, the video or image data may be indeterminate. Therefore, in one example, surgical hub 2404 can be further configured to compare physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively connected to surgical hub 2404) with visual or image data of hemostasis (e.g., from medical imaging device 124 communicatively connected to surgical hub 2404). Figure 2 The comparison is used to determine the integrity of sutures or tissue welds. In other words, the situational awareness system of the surgical hub 2404 can take physiological measurement data into account to provide additional context when analyzing visualization data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0248] Another benefit includes the proactive and automated control of the paired modular device 2402 based on specific steps of the surgical procedure being performed, reducing the number of times medical personnel need to interact with or control the surgical system 2400 during the surgical procedure. For example, if the situation-aware surgical hub 2404 determines that a subsequent step of the procedure requires the use of an RF electrosurgical instrument, it can proactively activate a generator connected to that instrument. Proactive activation of the power source allows the instrument to be ready for use immediately after the previous steps of the procedure have been completed.

[0249] As another example, the situation-aware surgical hub 2404 can determine whether the current or subsequent steps of the surgical procedure require a different view or magnification on the display based on one or more features(s) that the surgeon expects to view at the surgical site. The surgical hub 2404 can then proactively change the displayed view accordingly (e.g., provided by a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0250] As yet another example, the situation-aware surgical hub 2404 can determine which step of a surgical procedure is being performed or will be performed subsequently, and whether that step of the procedure requires specific data or comparisons between data. The surgical hub 2404 can be configured to automatically invoke a data screen based on the step of the surgical procedure being performed, without waiting for the surgeon to request that specific information.

[0251] Another benefit includes error detection during the setup of a surgical procedure or during the procedure itself. For example, a situational-aware surgical hub 2404 can determine whether the operating room is correctly or optimally set up for a surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgery being performed, retrieve (e.g., from memory) a corresponding list, product location, or setup requirement, and then compare the current operating room layout with a standard layout determined by the surgical hub 2404 for that type of surgery. In one example, the surgical hub 2404 can be configured to compare a list of items for the procedure (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 2404 with a suggested or anticipated list of items and / or devices for a given surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating the absence of a specific modular device 2402, patient monitoring device 2424, and / or other surgical items if any discontinuities exist between the lists. In one example, the surgical hub 2404 may be configured to determine, for example, the relative distance or position of the modular device 2402 and the patient monitoring device 2424 via a proximity sensor. The surgical hub 2404 may compare the relative position of the devices with a suggested or anticipated layout for a particular surgical procedure. The surgical hub 2404 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the suggested layout if any discontinuity exists between the layouts.

[0252] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from the intended procedure during a surgical operation. For instance, the surgical hub 2404 can be configured to determine the type of surgery being performed, retrieve (e.g., from memory) a corresponding list of steps or the order of equipment used, and then compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment determined by the surgical hub 2404 for that type of surgery. In one example, the surgical hub 2404 can be configured to provide an alarm indicating that an unexpected action is being performed or an unexpected device is being used at a specific step in the surgical procedure.

[0253] Overall, the situational awareness system used in the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (such as adjusting for different tissue types) and verifying actions during surgery. The situational awareness system also enhances the efficiency of surgeons performing surgical procedures by automatically suggesting the next step, providing data, and adjusting displays and other modular devices 2402 in the operating room, based on the specific context of the surgery.

[0254] See now Figure 21 It shows depictions of hubs such as surgical hubs 106 or 206 ( Figures 1 to 11 The situational awareness timeline 2500 is an illustrative surgical procedure and background information that surgical hubs 106 and 206 can derive from data received from the data source at each step of the surgical procedure. Timeline 2500 depicts the typical steps that nurses, surgeons, and other medical personnel will take during a segmentectomy, from setting up the operating room to transferring the patient to the postoperative recovery room.

[0255] Situational awareness surgical hubs 106 and 206 receive data from data sources throughout the surgical procedure, including data generated each time medical personnel utilize the modular devices paired with the surgical hubs 106 and 206. The surgical hubs 106 and 206 can receive this data from paired modular devices and other data sources, and continuously derive inferences about the ongoing surgery (i.e., background information) as new data is received, such as which step of the surgery is being performed at any given time. The situational awareness system of the surgical hubs 106 and 206 is capable, for example, recording data related to the process used to generate reports, verifying the steps being taken by medical personnel, providing data or cues that may be related to specific procedural steps (e.g., via a display screen), adjusting modular devices based on the background (e.g., activating monitors, adjusting the field of view (FOV) of medical imaging devices, or changing the energy level of ultrasound surgical instruments or RF electrosurgical instruments), and taking any other such actions described above.

[0256] As a first step 2502 in this exemplary procedure, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, surgical hubs 106 and 206 determine that the procedure to be performed is a thoracic surgery.

[0257] In the second step 2504, the staff scans the medical supplies to be brought in for the surgery. Surgical hubs 106 and 206 cross-reference the scanned supplies with a list of supplies used in various types of surgery and confirm that the supplied mixture corresponds to a thoracic surgery. Additionally, surgical hubs 106 and 206 can also determine if the surgery is not a wedge surgery (because the brought-in supplies lack certain supplies required for a thoracic wedge surgery, or are otherwise not corresponding to a thoracic wedge surgery).

[0258] In the third step 2506, medical personnel scan the patient band via a scanner communicatively connected to the surgical hubs 106 and 206. The surgical hubs 106 and 206 can then identify the patient based on the scanned data.

[0259] In step 4, 2508, medical staff activate assistive devices. The assistive devices used can vary depending on the type of surgery and the techniques the surgeon intends to use, but in this exemplary case, they include a fumigator, a blower, and a medical imaging device. Upon activation, as part of its initialization process, the assistive devices, as modular devices, can automatically pair with surgical hubs 106, 206 located in a specific vicinity of the modular device. Surgical hubs 106, 206 can then derive background information about the surgery by detecting the type of modular device paired with them during this preoperative or initialization phase. In this specific example, surgical hubs 106, 206 determine that the surgery is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies used in the surgery, and the type of modular device connected to the hub, surgical hubs 106, 206 can generally infer the specific procedure the surgical team will perform. Once the surgical hubs 106 and 206 know what specific surgery is being performed, they can retrieve the steps of the surgery from memory or the cloud and then cross-reference them with data subsequently received from connected data sources (e.g., modular devices and patient monitoring devices) to infer which step of the surgical procedure the surgical team is performing.

[0260] In step 5, 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices are compatible with surgical hubs 106 and 206. When surgical hubs 106 and 206 begin receiving data from the patient monitoring devices, they thus confirm that the patient is in the operating room.

[0261] Step 6, 2512: Medical personnel induce anesthesia in the patient. Surgical hubs 106 and 206 can infer that the patient is under anesthesia based on data from the modular device and / or patient monitoring devices (including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof). Upon completion of step 6, 2512, the preoperative portion of the lung segmental resection surgery is completed, and the surgical portion begins.

[0262] Step 7, 2514: Fold the patient's lung being operated on (while simultaneously switching ventilation to the contralateral lung). For example, surgical hubs 106 and 206 can infer from ventilator data that the patient's lung has collapsed. Surgical hubs 106 and 206 can infer that the surgical portion of the procedure has begun because they can compare the detection of lung collapse with the expected steps of the procedure (which can be previously accessed or retrieved), thereby determining that collapsing the lung is the first surgical step in that particular procedure.

[0263] Step 8, 2516: Insert a medical imaging device (e.g., an endoscope) and activate the video from the medical imaging device. Surgical hubs 106 and 206 receive data (i.e., video or image data) from the medical imaging device via their connection. After receiving the data, surgical hubs 106 and 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, surgical hubs 106 and 206 can determine that the specific procedure being performed is a segmental resection, not a lobectomy (note that wedge resection has been excluded based on data received by surgical hubs 106 and 206 at step 2504 of the procedure). From medical imaging device 124 ( Figure 2The data can be used to determine contextual information relevant to the type of surgery being performed in a variety of different ways, including by determining the angle of the visualization orientation of the medical imaging device relative to the patient's anatomy, monitoring the number of medical imaging devices used (i.e., those activated and paired with surgical hubs 106, 206), and monitoring the type of visualization device used. For example, a technique for performing a VATS lobectomy places the camera in the lower anterior corner of the patient's thoracic cavity above the diaphragm, while a technique for performing a VATS segmental resection places the camera in the anterior intercostal position relative to the segmental fissure. For example, the situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to identify the positioning of the medical imaging device based on the visualization of the patient's anatomy. As another example, a technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmental resection utilizes multiple cameras. As yet another example, a technique for performing a VATS segmental resection utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure not used in VATS lobectomy. By tracking any or all of this data from medical imaging devices, surgical hubs 106, 206 can thus determine the specific type of surgery being performed and / or the technique used for a particular type of surgery.

[0264] Step 9, 2518, involves the surgical team initiating the anatomical step of the procedure. Surgical hubs 106 and 206 can infer that the surgeon is dissecting to access the patient's lungs because they receive data from an RF generator or ultrasound generator indicating the firing of an energy device. Surgical hubs 106 and 206 can cross-reference the received data with the surgical procedure's retrieval steps to determine which point in the process (i.e., after the previously discussed surgical steps have been completed) corresponds to the anatomical step. In some cases, the energy device may be a power tool for the robotic arm mounted to the robotic surgical system.

[0265] Step 10, 2520: The surgical team continues with the ligation step of the surgery. Surgical hubs 106 and 206 can infer that the surgeon is ligating arteries and veins because they receive data from the surgical suture and cutting instruments indicating that the instruments are being fired. Similar to previous steps, surgical hubs 106 and 206 can deduce this inference by cross-referencing the data received from the surgical suture and cutting instruments with the retrieval steps in this process. In some cases, the surgical instruments may be surgical tools mounted on the robotic arm of a robotic surgical system.

[0266] Step 11, 2522: Performing the segmental resection portion of the surgery. Surgical hubs 106 and 206 can infer that the surgeon is transecting soft tissue based on data from surgical suture and cutting instruments (including data from their chambers). Chamber data may correspond to, for example, the size or type of staples fired by the instruments. Since different types of staples are used for different types of tissue, the chamber data can indicate the type of tissue being sutured and / or transected. In this case, the type of staple fired is used for soft tissue (or other similar tissue type), which allows surgical hubs 106 and 206 to infer that the segmental resection portion of the surgery is underway.

[0267] In step 12, 2524, the node dissection step is performed. Surgical hubs 106 and 206 can infer that the surgical team is dissecting a node and performing a leak test based on data received from the generator indicating that an RF or ultrasound instrument is being fired. For this particular surgery, the RF or ultrasound instrument used after transverse soft tissue incision corresponds to the node dissection step, which allows surgical hubs 106 and 206 to make such inferences. It should be noted that surgeons periodically switch between surgical suture / cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the specific steps in the surgery, as different instruments are better suited to specific tasks. Therefore, a specific sequence in which suture / cutting instruments and surgical energy instruments are used can indicate the steps of the surgery being performed by the surgeon. Furthermore, in some cases, robotic tools may be used for one or more steps in the surgery, and / or handheld surgical instruments may be used for one or more steps in the surgery. One or more surgeons may, for example, alternate between robotic tools and handheld surgical instruments and / or use the devices simultaneously. Upon completion of step 12, 2524, the incision is closed and the postoperative portion of the surgery begins.

[0268] Step 13, 2526: Reverse anesthesia of the patient. For example, surgical hubs 106, 206 can infer that the patient is waking up from anesthesia based on, for example, ventilator data (i.e., the patient's respiratory rate begins to increase).

[0269] Finally, step fourteen, 2528, involves medical personnel removing various patient monitoring devices from the patient. Therefore, when the hub loses EKG, BP, and other data from the patient monitoring devices, surgical hubs 2106 and 2236 can infer that the patient is being transferred to the recovery room. As can be seen from the description of this exemplary procedure, surgical hubs 2106 and 2236 can determine or infer when each step of a given surgical procedure occurs based on data received from various data sources communicatively connected to the surgical hubs 2106 and 2236.

[0270] Situational awareness is further described in various U.S. patent applications, which are incorporated herein by reference, and this disclosure is also incorporated herein by reference. In certain circumstances, the operation of a robotic surgical system (including, for example, the various robotic surgical systems disclosed herein) may be based on the situational awareness of hubs 2106 and 2236 and / or feedback from their components and / or from cloud 2104 ( Figure 17 It uses information to control [the system].

[0271] Imaging systems utilizing fusion imaging

[0272] Surgeons rely on imaging systems 142 Figure 2 One inherent problem with surgical procedures is the use of camera 144 ( Figure 2 Obstacles that impede the imaging system 142's ability to visualize the surgical site, thereby affecting the surgeon's ability to perform the surgical tasks required to operate. Obstacles may include, for example, fluids (e.g., blood) present at the surgical site, such as on the lens of camera 144, on the surface of a body cavity, or otherwise present at the surgical site; smoke generated by electrosurgical instruments or other aerosols present within the body cavity; and / or tissue or other structures covering the target tissue or structure. The surgical system may be configured to compensate for obstacles using various imaging techniques, including multispectral imaging of sub-regions of the camera 144's field of view (FOV), insertion of sub-regions from previously captured image frames by camera 144, contrastive multispectral analysis of captured images, etc.

[0273] In one general aspect, this disclosure relates to a surgical system configured to minimize visualization impairment due to various obstacles (e.g., surgical fumes) by utilizing image fragments captured at a certain sampling rate by an imaging system 142 using a multispectral light source. In one aspect, the surgical system may be configured to combine hyperspectral imaging with visible light imaging to minimize image interference caused by obstacles. For example, the surgical system may be configured to detect aspects of surgical instruments, surgical sites, or surrounding environments below or obscured by utilizing individual wavelengths or wavelength ranges of EMR. For example, the surgical system may utilize frames from sequential scanning devices to transmit individual wavelengths of EMR, including hyperspectral imaging devices configured to scan within and outside the visible spectrum, or second imaging systems configured to transmit EMR at a length different from that of a first or primary imaging system. Thus, the surgical system may be configured to identify obscured portions of image frames at specific wavelengths or a set of wavelengths and insert or replace portions of the obscured image portions using unobscured image portions of image frames obtained at other EMR wavelengths to provide the user with an unobscured image of the surgical site with full visualization.

[0274] In the use of imaging systems that include hyperspectral imaging devices, the hyperspectral imaging devices can scan at a specific rate (e.g., 240 frames per second), which allows a portion of the emitted scan to include EMR from near-infrared (IR) or ultraviolet (UV) laser sources. Because EMR at these wavelengths is not affected by obstacles (such as surgical fumes, liquids, etc.) as it is by visible light, hyperspectral imaging devices can be used to acquire images of shapes, contours, or features present in both hyperspectral and corresponding visible light images. The control system of the surgical system (e.g., Figure 2 The control system 133 shown can be configured to replace the obscured portion of an image obtained using visible light with a corresponding detected hyperspectral feature or image portion to achieve visualization of the surgeon. As another example, the imaging system may include a tunable EMR source (e.g., a spectral light source 150) that can be controlled by the control system 133 to emit EMR at wavelengths or wavelength groups where water absorption of EMR is minimal (e.g., in the visible blue-green wavelength range), as these are particularly susceptible to obstruction by water or aqueous fluids during surgery. As another example, in addition to the first primary imaging system, the surgical system may also include a second imaging system (e.g., Figure 2 The imaging system 142 shown is illustrated. In this example, the first imaging system 142 can be configured to image visible or near-visible EMR spectra, and the second imaging system can be configured to image different wavelength spectra (e.g., long-wavelength IR (LWIR)). Therefore, when the first imaging system is obstructed, the second imaging system can be activated by the surgical system or otherwise utilized as needed. In these different aspects, the surgical system will minimize the amount of cleaning required for camera 144 (e.g., removing obstructions from image sensor 135 or other scanning arrays) and prevent temporary obstruction of the surgical site due to obstacles between camera 144 and the surgical site (e.g., surgical smoke or blown gas).

[0275] In specific examples, imaging or visualization systems are described as including hyperspectral imaging devices or utilizing hyperspectral imaging techniques. However, it should be noted that hyperspectral imaging is a specific type of multispectral imaging. In hyperspectral imaging, the wavelength “library” is continuous, thus hyperspectral imaging techniques utilize the entire EMR spectrum. Conversely, multispectral can mean that the “library” is segmented. In other words, a multispectral imaging system can sense, for example, the visible, mid-wave IR (MWIR), and LWIR portions of the EMR spectrum (there may be gaps that the multispectral imaging system cannot sense, for example, the near-IR (NIR) portion of the EMR spectrum and / or between the MWIR and LWIR portions). The imaging or visualization systems and methods described herein should not be construed as limited to any specific example, including the example describing hyperspectral imaging. In fact, imaging or visualization systems and methods can widely utilize any multispectral imaging device and technique.

[0276] To aid in understanding the systems and methods described above, various examples will be described within the context of a video-assisted thoracoscopic surgery (VATS) procedure. It should be understood that this is for illustrative purposes only, and the systems and methods described are applicable to other contexts and / or surgical procedures. VATS is a surgical procedure in which one or more surgical instruments and one or more thoracoscopes (i.e., cameras) are inserted into the patient's thoracic cavity through an incision located between the patient's ribs. The cameras provide the surgeon with a view of the inside of the patient's thoracic cavity, allowing the surgeon to properly position / move the surgical instruments and manipulate the tissues / structures within the thoracic cavity. Therefore, Figure 22 This is a diagram of a surgical system 3000 during surgery on a lung 3010, according to at least one aspect of this disclosure. The surgical system 3000 for performing video-assisted surgery may include various surgical devices, including an imaging device 3002, a gripper 3004, an electrosurgical instrument 3006 or another surgical instrument, and a fume extractor 3008. Furthermore, the surgical system 3000 may include or be connected to surgical hubs 2106, 2236 (…). Figures 17-19 Visualization System 2108 Figures 17-19 ) or imaging system 142 ( Figure 2 ), control system 133 ( Figure 2 Robot System 2110 Figures 17-19 ) and any other systems or devices described herein. Imaging device 3002 may include camera 144 ( Figure 2 ), spectral light source 150 ( Figure 2 ), Structured light source 152 ( Figure 2 ), any other imaging transmitter or receiver described herein, or combinations thereof. Imaging device 3002 may be configured to capture images or videos of surgical sites within FOV 3020 and provide them to a display screen (e.g., Figure 2 The imaging device 3002 is configured to sense EMR within or outside the visible light portion of the EMR spectrum, thereby visualizing tissues and / or structures, whether visible or invisible to the naked eye. Based on the visualization provided by the imaging system 142 associated with the imaging device 3002, the surgeon can control surgical instruments to manipulate the tissues and / or structures to perform surgical procedures.

[0277] During surgical procedures, various obstructions (such as surgical smoke cloud 3014 or other aerosols, fluids, gases, tissues, structures, etc.) can move across the field of view 3020 of the imaging device 3002, preventing the imaging system 132 from fully visualizing the surgical site, which in turn negatively impacts the surgeon's ability to perform the procedure. Many surgical systems 3000 include a fume extractor 3008 to remove surgical smoke 3014, other aerosols, and gases from the body cavity where surgery is taking place. However, the fume extractor 3008 may be insufficient to remove all obstructions, or there may be delays associated with the removal of obstructions during which the surgeon cannot properly observe the surgical site. Therefore, systems and methods are needed to compensate for the presence of obstructions and allow observation of the surgical site through these obstructions.

[0278] In one respect, imaging systems (such as...) Figure 2 The imaging system 142 shown can be configured to allow visualization through obstructions using hyperspectral imaging and image fusion techniques. For example, Figure 23This is a diagram of an imaging device 3002 facing multiple obstructions. In this example, the surgical target is a subcutaneous tumor 3038. However, in order to actually visualize the tumor 3038, the imaging device 3002 will have to compensate for many different obstructions, including fluid 3030 present on the lens of the imaging device 3002, surgical smoke 3032 present in the body cavity, blood 3034 on the surface of tissue 3036, the tissue 3036 itself, and structures 3040 located throughout the tissue 3036. In one aspect, the imaging device 3002 may be a hyperspectral imaging device configured to sense EMR across a wavelength spectrum. EMR interacts differently with various objects at different wavelengths. In particular, a particular obstruction at a specific wavelength or wavelength range may not absorb certain wavelengths of EMR. Therefore, by sensing EMR in multiple portions of the EMR spectrum, the imaging system 142 can visualize through the obstruction by sensing EMR at wavelengths not absorbed by the obstruction. Furthermore, the wavelengths sensed by the imaging device 3002 can be selected to sense wavelengths that do not interact (or substantially do not interact) with typical or anticipated obstructions. In the depicted example, the imaging device 3002 can be configured to sense EMR within the visible, MWIR, and LWIR portions of the EMR spectrum.

[0279] In one aspect, the control system can be configured to visualize the surgical site in multiple portions of the EMR spectrum using multispectral (e.g., hyperspectral) imaging, and then provide the user with unmasked visualization by replacing the masked portion of an image captured in one wavelength range with the corresponding portion of an image captured in another wavelength range that is not absorbed by the mask. Figure 24 An example of such an algorithm is shown, which is a logic flowchart of process 3050 for generating a fused image using a multispectral EMR source. In the following description of process 3050, reference should also be made to... Figure 2 and Figure 23 Process 3050 may be implemented as, for example, instructions stored in memory 134 connected to control circuitry 132, which, when executed by control circuitry 132, cause control circuitry 132 to perform the enumerated steps of process 3050. For brevity, process 3050 is described as being executed by control circuitry 132; however, it should be understood that process 3050 may be executed by other combinations of hardware, software, and / or firmware.

[0280] Therefore, the control circuit 132 executing process 3050 can cause the imaging system 142 to perform step 3052 to sense the EMR (e.g., via imaging device 3002) from a first wavelength range (e.g., visible light) at the surgical site, and then perform step 3054 to generate a corresponding first image from it. Accordingly, the control circuit 132 can cause the imaging system 142 to perform step 3056 to sense the EMR (e.g., via imaging device 3002) from a second wavelength range (e.g., MWIR or LWIR) at the surgical site, and then perform step 3058 to generate a corresponding second image from it.

[0281] Therefore, control circuit 132 can perform step 3060 to determine whether the first image is at least partially occluded. Control circuit 132 can be configured to make this determination by detecting obstacles using object recognition and other computer vision techniques. If the first image is not at least partially occluded, process 3050 proceeds along the "No" branch, and control circuit 132 can continue with steps 3052 and 3054 to sense EMR, and perform steps 3054 and 3058 to generate the corresponding image, as described above. If the first image is at least partially occluded (i.e., an obstacle exists within the image), process 3050 proceeds along the "Yes" branch, and control circuit 132 can perform step 3062 to generate a third image by replacing the occluded portion of the first image with the corresponding portion of the second image. If a second wavelength range is selected such that it is not absorbed by the occluder, then the corresponding portion of the second image should be unoccluded. Therefore, the third image should provide an unoccluded visualization of the surgical site for the surgeon to view.

[0282] For simplicity, process 3050 is described in the context of generating and combining two images captured in two different wavelength ranges; however, imaging system 142 can be configured to sense and generate images in any number of wavelength ranges. For example, Figure 23An implementation is illustrated that combines image data from at least three different EMR wavelength ranges to generate a resulting image. Each of the depicted first image 3042a, second image 3042b, third image 3042c, and fourth image 3042d includes a pixel array 3043 that collectively visualizes a surgical site within a corresponding EMR wavelength range. In this example, the first image 3042a is captured using the visible light portion of the EMR spectrum and includes a first unobstructed portion 3044a, the remainder of which is obstructed; the second image 3042b is captured using the MWIR portion of the EMR spectrum and includes a second unobstructed portion 3044b; and the third image 3042c is captured using the LWIR portion of the EMR spectrum and includes a third unobstructed portion 3044c. The control system 133 can also be configured to perform various image processing techniques on the various generated images to improve the visualization thus provided. For example, the fourth image 3042d is also captured using the visible light portion of the EMR spectrum, and therefore can correspond to the first image 3042a, but includes additional image processing to identify the fluid (water) obstruction portion 3044d. Therefore, the corresponding portion of the first image 3042a can be filtered at the corresponding wavelength or wavelength range (e.g., the blue-green portion of the visible spectrum) to remove obstacles. Thus, the control circuit 132 of the execution process 3050 can be configured to generate a combined or fused image 3070 from the aforementioned initial images 3042a, 3042b, 3042c, and 3042d. The fused image 3070 may include a first portion 3072 corresponding to the unoccluded portion 3044a of a first image 3042a generated from the visible light portion of the EMR spectrum, a second portion 3074 corresponding to the unoccluded portion 3044b of a second image 3042b generated from the MWIR portion of the EMR spectrum, a third portion 3076 corresponding to the unoccluded portion 3044c of a third image 3042c generated from the LWIR portion of the EMR spectrum, and a fourth portion 3078 corresponding to the occluded portion 3044d of an image generated from the visible light portion of the EMR spectrum, but post-processed to remove the blue-green portion of the visible spectrum. Each of the above image portions 3072, 3074, 3076, and 3078 may be fused together by the control system 133 to generate the fused image 3070, which provides unoccluded visualization of the tumor 3038 and any other related structures 3040.

[0283] Another technique that can be used to compensate for occlusions present at surgical sites is image subregion interpolation, whereby occluded, damaged, or otherwise interfered image portions can be replaced by corresponding portions from a synchronized set of images. For example, a surgical control system can utilize Lucky Region Fusion (LRF) technology to improve the quality of visualization provided to the user by using multiple image frames. In one aspect, the control system can be configured to provide the user with unoccluded visualization by replacing occluded portions of an image with unoccluded portions of a previously captured image. Figure 26 An example of such an algorithm is shown, which is a logic flowchart of process 3100 for generating a fused image using multiple image frames. In the following description of process 3100, reference should also be made to... Figure 2 and Figures 27 to 29 Process 3100 may be implemented as, for example, instructions stored in memory 134 connected to control circuitry 132, which, when executed by control circuitry 132, cause control circuitry 132 to perform the enumerated steps of process 3100. For brevity, process 3100 is described as being executed by control circuitry 132; however, it should be understood that process 3100 may be executed by other combinations of hardware, software, and / or firmware.

[0284] Therefore, the control circuit 132 of execution process 3100 can (e.g., via imaging system 142) perform step 3102 to generate an image of the surgical site, and then perform step 3104 to determine whether the image is at least partially occluded, as described above. For example, Figure 27 This is a diagram of a series 3150 of n image frames 3160 captured by the imaging system 142. The nth image frame 3160 can be the most recently captured image frame 3160, the (n-1)th image frame 3160 can be a previously captured image frame 3160, and so on. For example, each image frame 3160 includes a plurality of pixels 3151, which may or may not correspond to pixels or units of the image sensor 135. Figure 27 As shown, image frame 3160 may include an unoccluded portion 3162 and an occluded portion 3164. Specifically, when evaluating the nth image frame 3160, the control circuit 123 of the execution process 3100 will determine that the nth image frame 3160 is at least partially occluded because the image frame includes the occluded portion 3164 of pixel 3151.

[0285] If control circuit 132 determines that image 3104 is not at least partially occluded, process 3100 proceeds along the "No" branch, and control circuit 132 can cause imaging system 142 to continue generating images that visualize the surgical site (i.e., visualize the surgical site), as described above. If control circuit 132 performs step 3104 to determine that the image is at least partially constructed (e.g., as shown in image frame 3160 of the nth image), process proceeds along the "Yes" branch, and control circuit 132 can perform step 3106 to retrieve previous images from image set 3150. In one aspect, control circuit 132 can perform step 3106 to continuously retrieve one or more previous images from image set 3150 until control circuit 132 has located the corresponding unoccluded image portions and replaced the occluded portions of the first image with these unoccluded image portions.

[0286] Therefore, control circuit 132 can perform step 3108 to generate an updated image from the original image and one or more previous images retrieved from image set 3150. For example, Figure 28 and Figure 29 An updated or fused image 3152 generated from multiple consecutive image frames 3160 is shown. In this particular example, n equals 60, although this is merely for illustrative purposes. Figure 28 In the figure, the number indicated within each pixel 3151 corresponds to the image frame 3160 from which the specific pixel 3151 is extracted. As shown, the fused image 3152 is generated from a combination of pixels 3151 spanning multiple different image frames 3160. Specifically, pixels 55 to 60 of image frames 3160 correspond sequentially to image frames (n-5) to nth, respectively. Figure 27 As shown. Therefore, the control circuit 132 can be configured to repeatedly perform step 3106 from the image set 3150 captured by the imaging system 142 to retrieve previous images and extract image portions (such as pixels 3151 that are not occluded in the retrieved image but correspond to pixels 3151 that are occluded in the subsequent images). The control circuit 132 can repeat this process until a set of fully or substantially unoccluded image portions has been retrieved from the image set 3150, and then the image portions are fused together in step 3108 to generate an updated image. Figure 29 The image shows the resulting fused image 3152 produced using this technique, illustrating how the tumor 3038 and structures 3140 (such as blood vessels) can be visualized to the user from an initially partially occluded image.

[0287] Another technique that can be used to compensate for obstructions present at the surgical site is to compare and analyze a set of simultaneous imaging devices. The control system 133 can be configured to interleave multiple image portions generated by multiple simultaneous imaging devices to generate a fused image. Specifically, the first or main imaging system (e.g., Figure 2 A portion of the image generated by the imaging system 142 shown can be replaced by a corresponding portion of the image generated by the second imaging system. Specifically, a set of imaging systems can be configured to time-index their scans. Obscured, damaged, blurred, or otherwise interfered portions of the first scan generated by the first imaging system can be replaced by clearer and / or verified portions of the second scan generated by the second imaging system (which are time-indexed based on the first scan). If image data is lost, damaged, or obscured in the imaging of the master dynamic dataset generated by the first imaging system, a secondary scan from another imaging system (which may also sense at another wavelength or wavelength range) can be used by the control system 133 to sharpen, replace, or interpolate the original image to improve the user's visualization of the surgical site.

[0288] Surgical system control based on multiple sensing parameters

[0289] A problem inherent in any surgical procedure and surgical instrument is controlling the instrument in an ideal manner for a given patient and / or tissue condition. To this end, some surgical instruments include sensors for sensing various parameters related to the instrument and / or the tissue manipulated by it. However, some sensed data can indicate different states or conditions of the tissue and may therefore be indeterminate in the absence of additional data. Therefore, surgical systems can combine data from imaging systems with other sensed data to resolve ambiguities and ideally control the surgical instrument based on the determined tissue state / condition.

[0290] In one general aspect, this disclosure relates to a control system configured to distinguish different states of tissue acted upon by surgical instruments using two related but distinct data sources. These states include, for example, fluid flow within the tissue and thermal shock to the tissue from energy directed by the surgical instruments. The control system may be configured to control the surgical instruments (such as surgical instrument 3290 described below).

[0291] Figure 30This is a schematic diagram of a surgical instrument 3290 configured to control various functions according to at least one aspect of the present disclosure. In one aspect, the surgical instrument 3290 is programmed to control distal translation of a displacement member such as a closure member 3264. The surgical instrument 3290 includes an end effector 3292, which may include a clamping arm 3266, a closure member 3264, and an ultrasonic scalpel 3268, which is interchangeable with or combined with one or more RF electrodes 3296 (shown in dashed lines). The ultrasonic scalpel 3268 is coupled to an ultrasonic transducer 3269 driven by an ultrasonic generator 3271.

[0292] In one aspect, sensor 3288 can be implemented as a limit switch, electromechanical device, solid-state switch, Hall effect device, MR device, GMR device, magnetometer, etc. In other embodiments, sensor 3288 can be implemented as a solid-state switch that operates under the influence of light, such as an optical sensor, IR sensor, ultraviolet sensor, etc. Similarly, the switch can be a solid-state device, such as a transistor (e.g., FET, junction FET, MOSFET, bipolar transistor, etc.). In other embodiments, sensor 3288 may include a conductorless switch, ultrasonic switch, accelerometer, and inertial sensor, etc.

[0293] In one aspect, the position sensor 3284 can be implemented as an absolute positioning system comprising an AS5055EQFT monolithic magnetic rotary position sensor, available from Austria Microsystems, AG. The position sensor 3284 can interact with control circuitry 3260 to provide the absolute positioning system. Position may include a Hall effect element located above the magnet and coupled to a CORDIC processor, also known for bitwise methods and Volder algorithms, provided to implement simple and efficient algorithms for calculating hyperbolic and trigonometric functions requiring only addition, subtraction, digital displacement, and table lookup operations.

[0294] In some examples, position sensor 3284 may be omitted. If motor 3254 is a stepper motor, control circuitry 3260 can track the position of closing member 3264 by aggregating the number and direction of steps the motor has been instructed to perform. Position sensor 3284 may be located in end effector 3292 or at any other part of the apparatus.

[0295] Control circuitry 3260 may communicate with one or more sensors 3288. Sensors 3288 may be positioned on end effector 3292 and adapted to operate with surgical instruments 3290 to measure various derived parameters, such as gap distance versus time, tissue compression versus time, and anvil strain versus time. Sensors 3288 may include magnetic sensors, magnetic field sensors, strain gauges, pressure sensors, force sensors, inductive sensors (such as eddy current sensors), resistive sensors, capacitive sensors, optical sensors, and / or any other suitable sensors for measuring one or more parameters of end effector 3292. Sensors 3288 may include one or more sensors.

[0296] An RF energy source 3294 is coupled to an end effector 3292, and the RF energy source is applied to the RF electrode 3296 when the RF electrode 3296 is disposed in the end effector 3292 to replace or in conjunction with the ultrasonic scalpel 3268. For example, the ultrasonic scalpel is made of a conductive metal and can serve as a return path for the electrosurgical RF current. A control circuit 3260 controls the delivery of RF energy to the RF electrode 3296.

[0297] Additional details are disclosed in U.S. Patent Application Serial 15 / 636,096, filed June 28, 2017, entitled “SURGICAL SYSTEM COUPLABLE WITHSTAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME”, the entire contents of which are incorporated herein by reference.

[0298] In various aspects, the sensor 3288 of the surgical instrument 3290 may include a sensor configured to detect or measure various electrical parameters related to the tissue on which the surgical instrument 3290 acts, such as the capacitance or impedance of the tissue. In various aspects, the sensor 3288 may also include a sensor configured to detect or measure various physical parameters related to the tissue on which the surgical instrument 3290 acts, such as temperature, viscoelastic compression (e.g., tissue creep, settling time, or initial loading rate) or thickness (e.g., which can be detected upon the first contact of the jaws with the tissue). Further, Figure 2The image sensor 135 of the control system 133 shown can be used to detect or measure various tissue parameters based on the EMR emitted by the imaging system 142 using the various techniques described above. For example, the image sensor 135 can be configured to detect the refractive index of the tissue at various wavelengths, the polarization of the EMR / light reflected by the tissue, the passive IR emission of the tissue, or the tissue-related Doppler wavelength shift. Any of these imaging-based parameters can be used in combination with other sensed parameters (e.g., electrical or physical parameters) to determine the state or condition of the tissue that may not be directly determined by the parameters alone.

[0299] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system based on the state or condition of the tissue to which the surgical instruments are acting, which can be determined based on parameters sensed by the imaging system and other sensors. Figure 31 An example of such an algorithm is shown, which is a logic flowchart of process 3300 for controlling a surgical system based on sensed parameters. In the following description of process 3300, reference should also be made to... Figure 2 and Figure 30 Process 3300 may be implemented as, for example, instructions stored in memory 134 connected to control circuitry 132, which, when executed by control circuitry 132, cause control circuitry 132 to perform the enumerated steps of process 3300. For brevity, process 3300 is described as being executed by control circuitry 132; however, it should be understood that process 3300 may be executed by other combinations of hardware, software, and / or firmware.

[0300] Therefore, the control circuit 132 of execution process 3300 can perform step 3302 via imaging system 142 to receive a measurement of the first tissue parameter. As previously described, the first tissue parameter may include, for example, the refractive index of the tissue at various wavelengths, the polarization of light reflected by the tissue, the passive IR emission of the tissue, or the tissue-related Doppler wavelength shift.

[0301] Therefore, control circuit 132 can perform step 3304 via sensor 788 to receive a measured value of the second tissue parameter. As previously described, the second tissue parameter may include, for example, various electrical and / or physical characteristics of the tissue, such as temperature, viscoelastic compression, or tissue thickness.

[0302] Therefore, control circuit 132 can perform step 3306 to determine the state or condition of the tissue based on a combination of the received tissue parameter measurements (steps 3302 and 3304), and then proceed to step 3308 to control surgical instrument 3290 accordingly. The same measurements of various electrical and / or physical characteristics of the tissue can indicate different tissue conditions, which in turn require different control adjustments to surgical instrument 3290. Without additional or supplementary information, when the tissue condition is unclear based on measurements of various electrical and / or physical characteristics, control system 133 may be unable to correctly control surgical instrument 3290 for a given tissue condition. Therefore, the control system described herein supplements the electrical and / or physical characteristics sensed by sensor 3288 with tissue parameters sensed via imaging system 142 to accurately determine the state or condition of the tissue and then control surgical instrument 3290 in an appropriate manner. For example, in response to the detection of an increase in tissue temperature at the end effector 3292 of the surgical instrument 3290 (i.e., the second tissue parameter received during process 3300 (step 3304)), different ways of controlling the surgical instrument 3290 may be appropriate. If the control system 133 detects a corresponding change in the polarization or refractive index of the tissue (i.e., the first tissue parameter received during process 3300 (step 3302)), the control circuit 132 may proceed to step 3306 to determine that the tissue has incidental thermal damage, and proceed to step 3308 to control the surgical instrument 3290 to reduce the instrument power level or provide a suggestion to the user to reduce the instrument power level. Conversely, if no corresponding change in the polarization or refractive index of the tissue is detected, the control circuit 132 may proceed to step 3306 to determine that the tissue does not have incidental thermal damage, and proceed to step 3308 to control the surgical instrument 3290 to maintain or increase the instrument power level or provide a suggestion to achieve such an effect. As another example, different ways of controlling the surgical instrument 3290 may be appropriate in response to detecting tissue impedance of the tissue grasped by the end effector 3292 of the surgical instrument 3290 (i.e., the second tissue parameter received during process 3300 (step 3304)). If the control system 133 does not detect a change in tissue impedance while the imaging system 142 visualizes the movement, peristalsis, or compression of the tissue (i.e., the first tissue parameter received during process 3300 (step 3302)), the control circuit 132 may proceed to step 3306 to determine the presence of subcutaneous irregularities in the grasped tissue.

[0303] The control system 133 described herein may be, for example, in surgical instrument 3290, surgical instrument 3290 (e.g., as Figure 21 The energy device 3241 shown can be communicatively connected to the surgical hub 2236 ( Figure 21The imaging data can be implemented on or executed by the imaging system 142 or a combination thereof (e.g., using a distributed processing protocol). When the control system is embodied as a component of the surgical instrument 3290, the imaging data can be received directly from the imaging system 142, or via the surgical hubs 2106, 2236. Figures 17-19 The surgical hub is connected to the imaging system 142. When the control system 133 is embodied as a component of the surgical hubs 2106 and 2236, it can receive imaging data from the imaging system 142 connected to the surgical hubs 2106 and 2236, and can receive surgical instrument sensor data from the surgical instruments 3290 connected to the surgical hubs 2106 and 2236. Then, the control system 133 of the surgical hubs 2106 and 2236 can determine appropriate surgical instrument control adjustments and transmit them to the surgical instruments 3290 for execution.

[0304] Adaptive optics system for compensating for imaging artifacts

[0305] In one aspect, control systems (such as combined) Figure 2 The described control system 133 can be configured to compensate for imaging artifacts associated with the imaging system 142 coupled thereto. In one aspect, the control system 133 can be configured to adjust the optical signals received by the imaging system 142 at multiple optical wavelengths within a sampling rate exceeding 60 Hz, incorporating selective imaging segment selection, to remove optical particle obstructions from the visualization. In another aspect, the control system 133 can be configured to emit a projected control beam (e.g., via the imaging system 142) and accordingly monitor the return signal on the isolation frame of the scanning array (e.g., image sensor 135) to determine the distortion of EMR / light by particles within the gas occupying the body cavity. The difference between the control projection and its source will provide a baseline for the control system 133, by which the range of visualization frames is adjusted in subsequent portions of the scan.

[0306] Surgical system control based on airborne particle characteristics

[0307] A problem inherent in surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke can include toxic gases and vapors; bioaerosols, including dead and living cell material, blood fragments, and viruses; and mutagenic and carcinogenic compounds. Therefore, it is essential to remove these particles from the surgical site, and accordingly, fumigators are commonly used in surgical procedures that generate surgical smoke. However, fumigators and other surgical devices (including surgical instruments) need to be controlled according to the type of particles generated, as different particle types may require different types of control adjustments to precisely control and mitigate smoke generation during surgery. Surgical systems can, for example, modify the energy profile of the surgical instruments to produce less smoke and / or automatically control the fumigator based on the type of particles generated.

[0308] In one general aspect, this disclosure relates to a control system configured to detect the polarization level of light emitted by an imaging system to determine parameters of a particle cloud and accordingly adjust control parameters of the linked system or apparatus. In another aspect, the polarization of the EMR reflected from the detected particles can be used in conjunction with the vectorization and number of generated particles to determine the source of the particles, which can in turn be used to control the apparatus generating the particles to improve visualization of the surgical site. In yet another aspect, the polarization of the EMR reflected from the detected particles can be used to determine whether adjusting the control parameters of an electrosurgical instrument or fumigator would more effectively improve visualization of the surgical site.

[0309] Figure 32 This is a diagram of a polarization EMR source 3500 for detecting different particle types according to at least one aspect of this disclosure. The polarization EMR source 3500 may include a transmitter 3502 configured to emit EMR 3506 and a polarization filter 3504 configured to polarize the emitted EMR 3506. The polarization filter 3504 may be removably or integrally attached to the transmitter 3502. The polarization EMR source 3500 may be embodied as a component of an imaging system, which may include, for example... Figure 1 The surgical visualization system 100 shown Figure 2 The imaging system 142 and / or shown Figure 5 The surgical visualization system 500 is shown. Accordingly, the transmitter 3502 may include... Figure 1 The transmitter 106 shown Figure 2 The structured light source 152 shown Figure 2 The spectral light source 150 shown is an example. The imaging system can also be considered a component of a surgical system, such as... Figure 1 The robotic surgical system 110 shown may further include a control system configured to control various aspects of the surgical system. For example, the control system may include… Figure 2 The control system 133 and / or shown Figure 11 The control system 600 shown is illustrated.

[0310] During surgical procedures, airborne particles may be present at the surgical site. These particles can include naturally occurring particles and non-natural or synthetic particles. Naturally occurring particles can be generated due to the interaction between surgical instruments (such as electrosurgical instruments) and the tissue being treated. Naturally occurring particles can include, for example, dead cell material, blood fragments, and other biological materials. Surgical personnel may introduce artificial or synthetic particles into the surgical site. These particles can manifest as smoke or aerosols present within or at the surgical site. Generally, the presence of such particles may be undesirable, therefore many surgical systems include fume extractors to remove unwanted smoke or aerosols from the surgical site. Imaging systems can be configured to detect particles (i.e., smoke) generated at the surgical site, and control systems can be configured to control various operating parameters of the surgical system or its components based on the characteristics or properties of the detected particles. This paper describes some examples of such control algorithms.

[0311] Return to reference Figure 32 As described above, airborne particles at the surgical site can include naturally occurring particles 3510 and artificial particles 3512. It is advantageous to be able to distinguish between different types of particles present at the surgical site because different actions may be required to reduce the presence of each different type of particle. For example, if the detected particle type is naturally occurring particle 3510, which can be generated by an electrosurgical instrument treating the tissue, it may be desirable to control the electrosurgical instrument to reduce the generation of naturally occurring particles 3510 (e.g., by reducing the energy duty cycle or otherwise altering the energy delivery profile of the instrument). Conversely, if the detected particle type is synthetic particle 3512, then controlling the electrosurgical instrument will have no effect on the presence of synthetic particle 3512, because this particle type is not generated by the action of the electrosurgical instrument. Instead, it may be desirable to increase the intake volume of the fumigator to remove synthetic particle 3512 from the surgical site. Furthermore, if a combination of different particle types is detected at the surgical site, it may be desirable to control the combined electrosurgical instrument and fumigator, where the control adjustments for each device are different. Therefore, control systems for surgical systems can be configured to detect different types of airborne particles present at the surgical site and to appropriately control various devices or components of the surgical system to reduce or eliminate particles at the surgical site.

[0312] In one aspect, naturally occurring particles 3510 and synthetic particles 3512 can be distinguished from each other based on the reflection characteristics of airborne particles 3510, 3512 when subjected to polarized EMR 3506. For example, control system 133 can be configured to enable transmitter 3502 to emit pulsed coherent EMR at multiple different wavelengths, both polarized and unpolarized, to determine the distance to the surgical body and define distance gating such that control system 133 performs depolarization measurements only from EMR 3506, which is reflected from particles within the air space between transmitter 3502 and the body cavity, rather than from the body cavity itself. Specifically, control system 133 can be configured to enable transmitter 3502 to pulsed coherent EMR at a first wavelength and a second wavelength. The first wavelength can be selected such that the EMR at the first wavelength does not substantially interact with naturally occurring particles 3510 and synthetic particles 3512, and therefore can penetrate smoke and reflect from the body cavity. Then, the control system 133 can determine the distance to the body cavity based on the time difference between the emitted EMR and the detected reflected EMR, using, for example, time-of-flight sensor systems 1104 and 1204, such as in combination with... Figures 14-16 The second wavelength can be selected such that the EMR at the second wavelength interacts substantially with both naturally occurring particles 3510 and synthetic particles 3512, thus allowing for the detection or measurement of characteristics associated with different particle types. Therefore, the control system 133 can utilize the cavity distance determined by pulsed EMR at the first wavelength to process the measurement results received by the EMR at the second wavelength, ensuring that only airborne particles are measured. The control system 133 can then determine whether the airborne particles are naturally occurring particles 3510 or synthetic particles 3512 based on the reflection characteristics of the airborne particles 3510, 3512, and control other components of the surgical system accordingly.

[0313] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system based on the type of airborne particles detected at the surgical site. Figure 33A An example of such an algorithm is shown; the diagram is a logic flowchart of a process 3600 for controlling a surgical system based on the detected particle type. Figure 33B This is a logic flowchart of process 3650, which controls the surgical system based on the type of particles detected within a defined range. In the following descriptions of processes 3600 and 3650, reference should also be made to... Figure 2Processes 3600 and 3650 may be implemented as, for example, instructions stored in a memory 134 connected to the control circuit 132, which, when executed by the control circuit 132, cause the control circuit 132 to perform the enumerated steps of processes 3600 and 3650. For brevity, processes 3600 and 3650 are described as being executed by the control circuit 132; however, it should be understood that processes 3600 and 3650 may be executed by other combinations of hardware, software, and / or firmware.

[0314] Now, let's get to the specifics. Figure 33A The control circuit 132 of the execution process 3600 can cause the imaging system 142 to perform step 3602 to emit polarized EMR directed toward the surgical site via, for example, a polarized EMR source 3500.

[0315] Therefore, control circuit 132 can perform step 3604 to receive polarized EMR reflected from airborne particles from a surgical site (e.g., within a body cavity), and step 3606 to determine whether the detected particle type is naturally occurring particle 3510 or artificial particle 3512. Control circuit 132 can distinguish between different types of airborne particles 3510, 3512 because these particles have different reflection characteristics when irradiated by polarized EMR. In particular, one type of airborne particle 3510, 3512 (e.g., artificial particle 3512) can scatter polarized EMR at a higher rate than the other type (e.g., naturally occurring particle 3510). This will reduce the visualization of the scattered airborne particle type, or otherwise affect how the reflected EMR is received by image sensor 135 of imaging system 142. Therefore, this difference in the visualization of different types of airborne particles 3510, 3512 can be characterized, and this difference can be used to identify the type of airborne particles 3510, 3512 present at the surgical site (e.g., within a body cavity).

[0316] Therefore, if the microparticle is naturally occurring microparticle 3510, process 3600 can proceed along the "yes" branch, and control circuit 132 can perform step 3608 to adjust the control parameters of the surgical system to a first state corresponding to naturally occurring microparticle 3510. Conversely, if the microparticle is artificial microparticle 3512, process 3600 can proceed along the "no" branch, and control circuit 132 can perform step 3610 to adjust the control parameters of the surgical system to a second state corresponding to artificial microparticle 3512.

[0317] On the other hand, there are various combinations of naturally occurring particles 3510 and artificial particles 3512 present in the body cavity. In this case, the control circuit 132 may alternatively determine the relative ratio of the types of airborne particles 3510, 3512 present in the body cavity (e.g., due to the relative degree of reduced or affected visibility) and then control the surgical device or combination of surgical devices accordingly.

[0318] On another front, the control system can be configured to perform distance gating on measurements and / or visualization based on the polarization EMR source 3500. This aspect can be achieved by... Figure 33B The process shown in the diagram is 3650.

[0319] Therefore, the control circuitry 132 of execution process 3650 can cause the imaging system 142 to perform step 3652 to emit EMR directed toward the surgical site at a first wavelength via, for example, a structured light source 152 and / or a spectral light source 150. In one aspect, the first wavelength may be a wavelength that does not substantially interact with the naturally occurring microparticles 3510 and synthetic microparticles 3512 to be imaged by the imaging system 142.

[0320] Therefore, the control circuit 132 can perform step 3654 to receive EMR reflected from the body cavity (i.e., surgical site) via the imaging system 142, and step 3656 to define a distance gating corresponding to the space between the transmitter of the imaging system 142 and the surface of the body cavity, as described above.

[0321] Therefore, control circuit 132 can cause imaging system 142 to perform step 3658 to emit polarized EMR directed toward the surgical site at a second wavelength via, for example, a polarized EMR source 3500. In one aspect, the second wavelength may be the wavelength that substantially interacts with the naturally occurring particles 3510 and synthetic particles 3512 to be imaged by imaging system 142.

[0322] Therefore, control circuit 132 can receive polarized EMR reflected within a defined distance gate via imaging system 142, which can correspond to airborne particles 3510, 3512 located between the transmitter of imaging system 142 and the body cavity surface. Then, control circuit 132 can perform step 3662 to determine whether the detected particle type is naturally occurring particle 3510 or artificial particle 3512, and perform steps 3608, 3610 to adjust the control parameters of the surgical system to a first state or a second state, as described above. Figure 33A The process described in 3600 is shown.

[0323] On the other hand, surgical systems can also be configured to track the movement of airborne particles throughout the surgical procedure, which can then be used to characterize the movement and size or structural changes of the cloud defined by the airborne particles. By characterizing the movement of the airborne particles over time, the surgical system can, for example, determine the degree of smoke extraction from the body cavity and then adjust or advise the user to adjust the location or amplitude of the smoke extraction or blowing. For example, the surgical system can deactivate a first smoke extractor or a first blower and activate a second smoke extractor or a second blower to regulate the gas circulation flow within the body cavity, thereby reducing any eddies within the body cavity (i.e., regions where the motion vector of particles and / or gases is zero or close to zero) to improve smoke extraction performance. As another example, the surgical system can adjust the level of the motor or fan of the smoke extractor or blower to improve smoke extraction performance.

[0324] In one aspect, the control system 133 can be configured to track and characterize the motion of airborne particles by characterizing the detection of particles on units or pixels of the image sensor 135. Specifically, the control system 133 can determine at which pixels the image sensor 135 detects particles, and then track the motion of the particles over time on the pixel array of the image sensor 135. In another aspect, the control system 133 can be configured to divide an image obtained via the image sensor 135 into two or more pixel array portions, generate motion vectors corresponding to a generalized change in position caused by the detected airborne particles from a first time point to a second time point, and then characterize the motion or configuration changes of the particle cloud accordingly.

[0325] For example, Figures 34A-34C A pixel array 3700 of an image sensor 135, consisting of multiple pixels 3701, is shown. Further, Figures 34A-34C The changes in the detected particle positions over time and the resulting generalized particle cloud motion vectors are shown. It should be noted that although the pixel array 3700 is depicted as a 5×5 array, this is for illustrative purposes only; the image sensor 135 or selected sub-portions of its pixels are not limited to a 5×5 array. Figures 34A to 34C In the following description, reference should also be made to Figure 2 .

[0326] Figure 34A The detection array 3700 at time t1 is shown, indicating that a first particle 3702a, a second particle 3702b, a third particle 3702c, a fourth particle 3702d, and a fifth particle 3702e are detected at a designated pixel 3701 of the image sensor 135. Figure 34BThe detection array 3700 at time t2 is shown, indicating that the positions of the first and fifth particles 3702a, 3702e have not changed, and the second, third, and fourth particles 3702b, 3702c, 3702d have been detected at different pixels 3701 of the image sensor 135. Based on the detected particle motion, the control circuit 132 can be configured to determine a vector representation of the motion of each pixel. The vector representation may include direction and magnitude. Based on the direction and magnitude of the motion vector, the control circuit 132 can be further configured to calculate (e.g., using vector addition) a vector 3704 corresponding to the generalized motion of the cloud defined by the detected particles. Thus, as Figure 35 As shown, based on the vector 3704 calculated from the pixel changes of the image sensor 135 that detects the particles, the control circuit 132 can track the changes of the particle cloud or aerosol from a first state 3710 (e.g., a first position or a first size) to a second state 3712 (e.g., a second position or a second size).

[0327] In one aspect, the control circuitry 132 coupled to the image sensor 135 can be configured to track the movement of detected airborne particles, calculate a generalized motion vector corresponding to the positional changes of the detected airborne particles within the pixel array 3700 (which may represent the entire pixel array of the image sensor 135 or a sub-part thereof), and then accordingly control various connected surgical devices (such as inhalers, fumigators, and / or surgical instruments). In another aspect, the control system 133 including the control circuitry 132 can be embodied as surgical hubs 2106, 2236 as described above under the heading "Surgical Hub Systems". In this aspect, surgical devices can be communicatively connected to (e.g., paired to) the surgical hubs 2106, 2236 and controlled according to the described system and process.

[0328] In another aspect, the control system 133 can be configured to utilize Raman spectroscopy to determine the vibrational / rotational aspects of gaseous particles using, for example, near-IR, UV, or a combination of near-IR and UV wavelengths. For example, data obtained from these techniques can provide information about the gaseous species (e.g., benzene and aldehydes), which in turn can provide insights into the tissue type that produced the particles or the energy efficiency applied to the tissue. The control system 133 may include, for example, filters (e.g., bandpass or notch filters) coupled to the detector to filter out elastic scattering from the source EMR, since desired information about the species is contained in the inelastic scattering of the EMR. The signal generated by the image sensor 135 or another such detector (e.g., a CCD detector) according to Raman spectroscopy can be based on the inherent structural characteristics of the detected molecule. In particular, Raman spectroscopy is based on the concept that, for example, photons emitted by a suitable emitter excite a molecule to a higher energy state, causing the scattered photons to change frequency due to energy savings from vibrational / rotational changes within the molecule. Depending on the specific type of monochromatic light source used, for a given excitation frequency, this variation in the frequency of scattered photons can be used to characterize the type of molecules with which the photons interact, by comparing the detected signal with pre-characterized data. The identified particle molecule types can be used in many different applications, including providing specific data on the relative amounts of potentially hazardous molecules generated at surgical sites for safety monitoring. The identified particle molecule types can also be used to evaluate the effectiveness and operational status of fume extractor systems or their filters.

[0329] Surgical system control based on smoke cloud characteristics

[0330] A problem inherent in surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke can include toxic gases and vapors; bioaerosols, including dead and living cell material, blood fragments, and viruses; and mutagenic and carcinogenic compounds. Therefore, it is essential to remove these particles from the surgical site, and accordingly, fumigators are commonly used in surgical procedures that generate surgical smoke. However, fumigators and other surgical devices (including surgical instruments) need to be controlled based on the amount of smoke at the surgical site, changes in the smoke cloud over time (e.g., whether the smoke cloud actively increases or decreases), and other characteristics of such smoke clouds, in order to precisely control and reduce smoke generation during surgery. Surgical systems can, for example, modify the energy profile of the surgical instruments to produce less smoke and / or automatically control the fumigator based on the amount of surgical smoke generated.

[0331] In one general aspect, this disclosure relates to a surgical system configured to detect and characterize an amorphous three-dimensional cloud of particles generated during surgical procedures. The surgical system can be configured to detect the movement of the particle cloud within the abdominal cavity and its movement relative to the surgical site, and then accordingly control various surgical devices (such as surgical instruments or smoke extractors). In another general aspect, this disclosure relates to a control system configured to define the surface or boundary of the cloud or particle cluster generated during surgical procedures and analyze various characteristics of the defined cloud (such as the orientation and rate of change of the boundary) to control various control parameters of the surgical system (such as the power level controlled by the surgical instrument / generator or smoke extractor motor). In yet another aspect, the control system can be configured to form a boundary by defining a predetermined density of particles based on the total volume of the particles or the size of the particles. In yet another aspect, the rate of change of the particle cloud surface boundary can be used to orient the rate of change of an energy device or smoke extractor.

[0332] Figure 36 This is a diagram of a surgical system 3750 during the execution of a surgical procedure generating a microparticle cloud 3752, according to at least one aspect of this disclosure. The surgical system 3750 may be embodied as a robotic surgical system, for example... Figure 1 The robotic surgical system 110 is shown. Surgical system 3750 may include electrosurgical instruments 3754, a fume extractor 3756, a gripper 3750, and any other surgical devices for handling, cutting, or otherwise manipulating tissue 3760 during surgical procedures. Although not explicitly stated... Figure 36 As shown, however, the surgical system 3750 may further include an imaging system, which may include, for example... Figure 1 The surgical visualization system 100 shown Figure 2 The imaging system 142 and / or shown Figure 5 The surgical visualization system 500 is shown. For example, the surgical system 3750 may also include a control system, which may include... Figure 2 The control system 133 and / or shown Figure 11 The control system 600 shown is illustrated.

[0333] During surgical procedures, airborne particles 3751 can be generated due to the interaction between surgical instruments 3754 (such as electrosurgical instruments) and the tissue being treated 3760. These particles 3751 can manifest as a cloud of smoke 3752 or aerosol present within or at the surgical site. Generally, the presence of such particles 3751 may be undesirable, and therefore many surgical systems 3750 include a fume extractor 3756 to remove particles 3751 from the surgical site. However, imaging systems can be configured to image the particles 3751 or smoke generated at the surgical site, and control systems can be configured to control various operating parameters of the surgical system 3750 or its components based on the characteristics or properties of the imaged smoke. Some examples of such control algorithms are described herein.

[0334] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system 3750 based on one or more characteristics associated with a smoke cloud generated at the surgical site. Figure 37 An example of such an algorithm is shown, which is a logic flowchart of process 3800 for controlling a surgical system based on smoke cloud characteristics. Reference should also be made to the following description of process 3800. Figure 2 Process 3800 may be implemented as, for example, instructions stored in memory 134 connected to control circuitry 132, which, when executed by control circuitry 132, cause control circuitry 132 to perform the enumerated steps of process 3800. For brevity, process 3800 is described as being executed by control circuitry 132; however, it should be understood that process 3800 may be executed by other combinations of hardware, software, and / or firmware.

[0335] Therefore, the control circuit 132 of execution process 3800 can use any of the above-described techniques to perform step 3802 to detect the presence of airborne particles within the FOV of imaging system 142. Typically, the image sensor 135 of imaging system 142 can detect EMR emitted by structured light source 152 and / or spectral light source 150 and reflected by airborne particles to detect particles / image particles.

[0336] Therefore, control circuit 132 can perform step 3804 to characterize the particle cloud defined by the detected particles. In one aspect, control circuit 132 can be configured to define the three-dimensional boundary of the particle cloud to depict an amorphous three-dimensional structure whose density, volume, position, movement, and / or boundaries can be tracked over time. The boundary of the particle cloud at the surgical site can be defined in various different ways. For example, the particle cloud boundary can be defined as the volume of all airborne particles detected within the FOV of imaging system 142. As another example, the particle cloud boundary can be defined as the volume having a threshold density of airborne particles.

[0337] Therefore, control circuit 132 can perform step 3806 to determine whether one or more characteristics of the particle cloud violate a threshold. Such tracking characteristics may include, for example, the density of the particle cloud, the volume of the particle cloud, the location of the particle cloud and / or its boundaries, the movement of the particle cloud and / or its boundaries, and / or the rate of change or other derivative of any of the above characteristics. The threshold used for tracking characteristics may be pre-programmed or depend on other parameters, such as the surgical context (e.g., the type of surgery being performed). If no threshold is violated, process 3800 can proceed along the "No" branch, and control circuit 132 can continue with the steps described above until, for example, a stopping criterion has been met (e.g., the surgery is completed). If the threshold is violated, process 3800 can proceed along the "Yes" branch, and control circuit 132 can continue with the steps described below.

[0338] Therefore, control circuit 132 can perform step 3808 to adjust one or more control parameters of surgical system 3750. Control parameters that can be adjusted by control circuit 132 may include surgical instrument / generator energy levels, exhaust suction, visualization parameters, etc. For example, Figure 38 These are a series of graphs 3850, 3852, and 3854, illustrating the adjustment of control parameters by the control circuit 132 of the execution process 3800 based on the characteristics of the particulate cloud. The first graph 3850 shows a first line 3860, indicating the smoke cloud density represented by the vertical axis 3856 and time represented by the horizontal axis 3858. The second graph 3852 shows a second line 3868, indicating the energy duty cycle of the electrosurgical instrument 3754 (or the generator driving the electrosurgical instrument 3754) represented by the vertical axis 3866 and time represented by the horizontal axis 3858. The third graph 3854 shows a third line 3880, indicating the change in the smoke exhaust or extraction flow rate of the smoke exhauster 3756 represented by the vertical axis 3878 and time represented by the horizontal axis 3858. Combined, graphs 3850, 3852, and 3854 illustrate a representative, predictive implementation of surgical procedure 3800, in which procedure 3800 performs step 3808 to adjust the electrosurgical instrument energy duty cycle and smoke exhauster suction flow rate control parameters based on the characterized smoke cloud density.

[0339] Initially, the electrosurgical instrument 3754 does not apply energy to the captured tissue 3760, as shown in the first figure 3890. Therefore, the energy duty cycle of the electrosurgical instrument 3754 is zero, the fumigator suction flow rate is at a base or default rate, and no smoke is generated (because no energy is applied to the tissue 3760). At time t1, the surgeon activates the electrosurgical instrument 3754 and begins to apply energy to the tissue 3760, represented by line 3870, where the energy duty cycle increases from 0 to E3. Due to the application of energy to the tissue 3760, smoke begins to be generated at the surgical site, a process represented by line 3862, where the smoke cloud density increases sharply from zero over a period after t1. Further, in response to the activated energy, as the fumigator 3756 begins to attempt to remove the generated smoke from the surgical site, the flow rate of the fumigator can be increased from Q1 to Q2 via control circuit 132 (as shown by line 3882). At this stage, the control circuit 132 may perform step 3802 to begin detecting airborne particles generated by the applied energy, and step 3804 to characterize the corresponding smoke cloud defined by the airborne particles.

[0340] At time t2, applying energy to tissue 3760 causes a smoke cloud 3752 to form at the surgical site, as shown in the second figure 3892. Control circuitry 132 can proceed to step 3806 to determine that the cloud density has exceeded a smoke cloud density threshold (e.g., represented by D3). Therefore, control circuitry 132 adjusts this parameter by decreasing the electrosurgical instrument energy duty cycle control parameter from E3 to E2 (as shown in line 3872) to proceed to step 3808. Control circuitry 132 can optionally make this adjustment because applying a lower level of energy to tissue 3760 results in less smoke. In response, the smoke cloud density begins to decrease at time t1 (as shown in line 3864).

[0341] At time t3, the size of smoke 3752 has decreased, but it has not yet completely dissipated, as shown in the third graph 3894. Control circuit 132 can proceed to step 3806 to determine if the cloud density has not decreased at a sufficiently fast rate, or if some other characteristic of the smoke cloud has violated some other threshold. Therefore, control circuit 132 again proceeds to step 3808 to adjust the parameter by reducing the electrosurgical instrument energy duty cycle control parameter from E2 to E1 (as shown in line 3874) in order to mitigate further smoke generation.

[0342] At time t4, the smoke 3752 has almost dissipated, as shown in the fourth figure 3896. Control circuit 132 can proceed to step 3806 to determine that the smoke cloud has violated another threshold, such as a cloud density exceeding a certain level (e.g., as indicated by D1) for a longer period than the threshold time interval (e.g., as shown by t4). Therefore, control circuit 132 adjusts this by increasing the exhaust fan suction flow rate control parameter from Q2 to Q3 (as shown by line 3884) to completely remove smoke particles from the surgical site.

[0343] It should be noted that Figure 38 The implementation of process 3800 shown is provided for illustrative purposes and represents only one possible implementation. Specifically, different control parameters can be controlled through process 3800, different thresholds can be utilized, and different smoke cloud characteristics can be tracked, etc. Therefore, Figure 38 It should not be interpreted as restricting in any way. Figure 37 The process 3800 or any other system and method described herein.

[0344] Exemplary clinical applications

[0345] The various surgical visualization systems disclosed herein can be used in one or more of the following clinical applications. The following clinical applications are non-exhaustive and merely illustrative applications of one or more of the various surgical visualization systems disclosed herein.

[0346] The surgical visualization system disclosed herein can be used in a variety of surgical procedures for various medical specialties, such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, obesity / gastroenterology, and hepatobiliary surgery (HPB). For example, in urological surgeries (such as prostatectomy), the ureter can be detected in fat, or connective tissue and / or nerves can be detected in fat. Similarly, in gynecological oncology surgeries (such as hysterectomy) and colorectal surgeries (such as low anterior resection (LAR)), the ureter can be detected in fat and / or connective tissue. In thoracic surgeries (such as lobectomy), blood vessels can be detected in the lungs or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In obesity surgeries, blood vessels can be detected in fat. For example, in HPB procedures (such as hepatectomy or pancreatectomy), blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0347] In one example, a clinician might want to remove an endometrial fibroid. Based on a preoperative magnetic resonance imaging (MRI) scan, the clinician knows that the fibroid is located on the surface of the intestine. Therefore, the clinician might want to know intraoperatively which tissues constitute part of the intestine and which constitute part of the rectum. In such cases, a surgical visualization system, as disclosed herein, can indicate the different types of tissues (intestine and rectum) and convey this information to the clinician via an imaging system. Furthermore, the imaging system can determine and transmit the proximity of the surgical apparatus to the selected tissue. In such cases, a surgical visualization system can provide increased surgical efficiency without serious complications.

[0348] In another example, clinicians (e.g., gynecologists) may keep away from certain anatomical areas to avoid getting too close to critical structures, and therefore may be unable to remove, for example, all endometriosis. Surgical visualization systems, as disclosed herein, allow gynecologists to reduce the risk of getting too close to critical structures, enabling them to use surgical instruments close enough to remove all endometriosis, which improves patient outcomes (democratizing surgery). Such systems allow surgeons to “keep moving” during surgery rather than repeatedly stopping and starting in order to identify areas to avoid, especially during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and given the presentation and / or thickness of the tissues involved, the system may be particularly useful for hysterectomy and endometriosis removal.

[0349] In another example, clinicians may risk dissecting blood vessels too close to the target lobe, potentially affecting blood supply to lobes other than the target lobe. Furthermore, patient-to-patient anatomical differences may lead to dissections based on the specific patient, affecting blood vessels (e.g., branches) in different lobes. Surgical visualization systems as disclosed herein enable the identification of the correct blood vessels at the desired location, allowing clinicians to perform dissections with appropriate anatomical certainty. For example, the system can confirm the correct blood vessel is in the correct location, and the clinician can then safely separate the vessel.

[0350] In another example, due to the uncertainty of the anatomical structure of blood vessels, clinicians may perform multiple dissections before reaching the optimal location. However, in the first case, it is desirable to perform the dissection at the optimal location, as more dissections can increase the risk of bleeding. Surgical visualization systems, as disclosed herein, can minimize the number of dissections by indicating the correct blood vessels and the optimal location for dissection. For example, the ureter and cardinal ligament are densely packed and present unique challenges during dissection. In such cases, minimizing the number of dissections may be particularly desirable.

[0351] In another example, a clinician (e.g., a surgical oncologist removing cancerous tissue) may want to know the identification of key structures, the location of the cancer, the stage of the cancer, and / or an assessment of tissue health. This type of information goes beyond what a clinician can see with the naked eye. Surgical visualization systems, as disclosed herein, can identify and / or communicate this information to the clinician intraoperatively to enhance intraoperative decision-making and improve surgical outcomes. In some cases, surgical visualization systems may be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, such as those using endoscopy or exoscopy.

[0352] In another example, a clinician (e.g., a surgical oncologist) might want to disable one or more alerts regarding the proximity of surgical instruments to one or more critical structures to avoid being overly conservative during surgery. In other cases, clinicians may want to receive certain types of alerts, such as tactile feedback (e.g., vibration / beep), to indicate proximity and / or “no-fly zones” to maintain adequate distance from one or more critical structures. For example, surgical visualization systems, as disclosed herein, can provide flexibility based on the clinician’s experience and / or the expected aggressiveness of the procedure. In such cases, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. Surgical visualization systems can aid in planning subsequent steps during surgery.

[0353] Various aspects of the subject matter described herein are set forth in the following numbered embodiments.

[0354] Example 1. A control system for a surgical system, the control system including an imaging system and control circuitry coupled to the imaging system. The imaging system includes a transmitter configured to emit multispectral electromagnetic radiation (EMR). The control circuitry is configured to detect airborne particles via the emitted EMR through the image sensor, characterize the boundary of a particle cloud containing the detected airborne particles, and adjust control parameters of the surgical system according to the characteristics of the boundary of the particle cloud.

[0355] Example 2. The control system according to Example 1, wherein the control circuit is further configured to detect the characteristic change of the boundary of the particle cloud and adjust the control parameter from a first value to a second value according to the characteristic change.

[0356] Example 3. The control system according to Example 1 or 2, wherein the surgical system includes a generator configured to drive electrosurgical instruments, and the control parameters include the energy duty cycle of the generator.

[0357] Example 4. The control system according to Example 1 or 2, wherein the surgical system includes a smoke exhauster, and the control parameters include at least one of the fan state or motor state of the smoke exhauster.

[0358] Example 5. The control system according to any one of Examples 1 to 4, wherein the characteristics of the boundary include the direction of movement of the boundary.

[0359] Example 6. The control system according to any one of Examples 1 to 4, wherein the characteristics of the boundary include the rate of change of the boundary.

[0360] Example 7. A control system for a surgical system, the control system including an imaging system and control circuitry coupled to the imaging system. The imaging system includes a multispectral electromagnetic radiation (EMR) source and an image sensor. The control circuitry is configured to detect EMR reflected from particles in an aerosol and received by the image sensor, characterize the configuration of the aerosol containing the detected particles, and adjust at least one of a first control parameter or a second control parameter of the surgical system based on the characterized configuration of the aerosol.

[0361] Example 8. The control system according to Example 7, wherein the control circuit is further configured to detect changes in the configuration of the aerosol and adjust at least one of the first control parameter or the second control parameter from a first value to a second value according to the changes in configuration.

[0362] Example 9. The control system according to Example 7 or 8, wherein the surgical system includes a generator configured to drive an electrosurgical instrument, and the first control parameter includes the energy duty cycle of the generator.

[0363] Example 10. The control system according to Example 7 or 8, wherein the surgical system includes a smoke exhauster, and the second control parameter includes at least one of the fan state or motor state of the smoke exhauster.

[0364] Example 11. The control system according to any one of Examples 7 to 10, wherein the configuration includes the direction of movement of the aerosol.

[0365] Example 12. The control system according to any one of Examples 7 to 10, wherein the configuration includes the rate of change of the aerosol.

[0366] Example 13. A method for controlling a surgical system, the surgical system including an imaging system comprising a transmitter configured to emit multispectral electromagnetic radiation (EMR) and an image sensor. The method includes the steps of: detecting airborne particles via the image sensor based on the emitted EMR, characterizing the boundary of a particle cloud containing the detected airborne particles, and adjusting control parameters of the surgical system according to the characteristics of the boundary of the particle cloud.

[0367] Example 14. The method according to Example 13, wherein the control circuit is further configured to detect the feature change of the boundary of the particle cloud and adjust the control parameter from a first value to a second value according to the feature change.

[0368] Example 15. The method according to Example 13 or 14, wherein the surgical system includes a generator configured to drive an electrosurgical instrument, and the control parameters include the energy duty cycle of the generator.

[0369] Example 16. The method according to Example 13 or 14, wherein the surgical system includes a smoke extractor, and the control parameters include at least one of the fan state or motor state of the smoke extractor.

[0370] Example 17. The method according to any one of Examples 13 to 16, wherein the characteristics of the boundary include the direction of movement of the boundary.

[0371] Example 18. The method according to any one of Examples 13 to 16, wherein the characteristics of the boundary include the rate of change of the boundary.

[0372] Although several forms have been illustrated and described, the applicant does not intend to limit or restrict the scope of the appended claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be made without departing from the scope of this disclosure, and those skilled in the art will recognize such modifications, variations, alterations, substitutions, combinations, and equivalents. Furthermore, alternatively, the structure of each element associated with a described form can be described as a device for providing the function performed by said element. Additionally, where materials for certain components are disclosed, other materials may also be used. Therefore, it should be understood that the foregoing detailed descriptions and the appended claims are intended to cover all such modifications, combinations, and variations falling within the scope of the forms disclosed in this invention. The appended claims are intended to cover all such modifications, variations, alterations, substitutions, modifications, and equivalents.

[0373] The specific embodiments described above have illustrated various forms of apparatus and / or methods using block diagrams, flowcharts, and / or examples. Wherever such block diagrams, flowcharts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flowcharts, and / or examples can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be equivalently implemented in an integrated circuit, wholly or partially, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as any combination thereof, and that designing circuit systems and / or writing software and / or hardware code according to this disclosure will be within the skill of those skilled in the art. Furthermore, those skilled in the art will recognize that the mechanisms of the subject matter described herein can be distributed as one or more program products in various forms, and that the exemplary forms of the subject matter described herein apply regardless of the specific type of signal-bearing medium used for actual distribution.

[0374] Instructions used for programming logic to execute various disclosed aspects may be stored in the system's memory, such as dynamic random access memory (DRAM), cache, flash memory, or other memory. Furthermore, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any means for storing or transmitting information in a machine-readable (e.g., computer-readable) form, but are not limited to floppy disks, optical disks, optical disc read-only memory (CD-ROM), and magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage devices used for transmitting information over the Internet via electrical signals, optical signals, acoustic signals, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, non-transitory computer-readable media include any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0375] As used in any aspect of this document, the term "control circuitry" may refer to, for example, hardwired circuitry systems, programmable circuitry systems (e.g., computer processors including one or more individual instruction processing cores, processing units, processors, microcontrollers, microcontroller units, controllers, digital signal processors (DSPs), programmable logic devices (PLDs), programmable logic arrays (PLAs), field-programmable gate arrays (FPGAs)), state machine circuitry systems, firmware storing instructions executed by the programmable circuitry system, and any combination thereof. Control circuitry can be implemented collectively or individually as part of a larger system, such as integrated circuits (ICs), application-specific integrated circuits (ASICs), system-on-a-chip (SoCs), desktop computers, laptop computers, tablet computers, servers, smartphones, etc. Therefore, as used herein, "control circuit" includes, but is not limited to, electronic circuits having at least one discrete circuit, electronic circuits having at least one integrated circuit, electronic circuits having at least one application-specific integrated circuit, electronic circuits forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program that at least partially implements the methods and / or devices described herein, or a microprocessor configured by a computer program that at least partially implements the methods and / or devices described herein), electronic circuits forming a memory device (e.g., forming a random access memory), and / or electronic circuits forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). Those skilled in the art will recognize that the subject matter described herein can be implemented in analog or digital modes, or some combination thereof.

[0376] As used in any aspect of this document, the term "logic" can refer to an application, software, firmware, and / or circuit system configured to perform any of the foregoing operations. Software can be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware can be embodied as hard-coded (e.g., non-volatile) code, instructions, or instruction sets and / or data in a memory device.

[0377] As used in any part of this document, the terms “component,” “system,” “module,” etc., can refer to computer-related entities, hardware, combinations of hardware and software, software, or software in execution.

[0378] As used in any aspect of this document, "algorithm" refers to a systematic sequence of steps that leads to a desired result, where "step" refers to the manipulation of physical quantities and / or logical states, which may (but not necessarily) take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities and are merely convenient labels applied to these quantities and / or states.

[0379] The network may include a packet-switched network. Communication devices may be able to communicate with each other using a selected packet-switched network communication protocol. An exemplary communication protocol may include an Ethernet communication protocol that may allow communication using Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" published by the Institute of Electrical and Electronics Engineers (IEEE) in December 2008 and / or a higher version of this standard. Alternatively or additionally, communication devices may be able to communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices may be able to communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol may conform to or be compatible with standards published by the International Telegraph and Telephone Consultative Committee (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may be able to communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard entitled "ATM-MPLS Network Interworking 2.0" and / or a higher version of that standard, published by the ATM Forum in August 2001. Of course, this document also envisions different and / or subsequently developed connectivity-oriented network communication protocols.

[0380] Unless otherwise expressly stated in the foregoing disclosure, it is understood that in the foregoing disclosure, discussions using terms such as “processing,” “estimating,” “calculating,” “determining,” and “displaying” refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities in the registers and memories of the computer system and convert them into other data similarly represented as physical quantities in the memory or registers of the computer system or other such information storage, transmission, or display devices.

[0381] One or more components may be referred to herein as “configured to be,” “configurable to be,” “operable / operationally,” “suitable / adaptable,” “capable,” “adaptable / fittable,” etc. Those skilled in the art will recognize that, unless the context otherwise requires, “configured to be” generally encompasses components in an active state and / or in an inactive state and / or in a standby state.

[0382] The terms "proximal" and "distal" are used herein in relation to the clinician manipulating the handle portion of the surgical instrument. "Proximal" refers to the portion closest to the clinician, and "distal" refers to the portion furthest from the clinician's position. It should also be understood that, for brevity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used in conjunction with accompanying drawings. However, surgical instruments are used in many orientations and locations, and these terms are not restrictive and / or absolute.

[0383] Those skilled in the art will recognize that, in general, the terminology used herein, and particularly in the appended claims (e.g., the text of the appended claims), is typically intended to be “open” terms (e.g., the term “comprising” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “at least having,” the term “including” should be interpreted as “comprising but not limited to,” etc.). Those skilled in the art will also understand that if a specific number of statements in the introduced claims is intended, such an intention will be explicitly stated in the claims, and if no such statement is present, such an intention does not exist. For example, to aid understanding, the appended claims below may contain the use of the introductory phrases “at least one” and “one or more” to introduce the claims. However, the use of such phrases should not be construed as implying that introducing a claim statement with the indefinite article "a" or "an" limits any particular claim containing such an introductory claim statement to a claim containing only one such statement, even when the same claim includes the introductory phrase "one or more" or "at least one" and indefinite articles such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); this also applies to the use of definite articles used to introduce a claim statement.

[0384] Furthermore, even when a specific number of claims is explicitly stated, those skilled in the art should recognize that such a statement should generally be interpreted as referring to at least the number stated (e.g., in the absence of other modifiers, a bare statement of "two statements" generally means at least two statements, or two or more statements). Moreover, in cases where conventions such as "at least one of A, B, and C" are used, such constructions are generally intended to have a meaning that those skilled in the art will understand (e.g., "a system having at least one of A, B, and C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In cases where conventions such as "at least one of A, B, or C" are used, such constructions are generally intended to have a meaning that those skilled in the art will understand (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art should also understand that, generally, unless the context otherwise indicates, any transitional words and / or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to cover the possibility of including one of the terms, any one of the terms, or both of the terms. For example, the phrase "A or B" will generally be understood to include the possibility of "A" or "B" or "A and B".

[0385] With respect to the appended claims, those skilled in the art will understand that the operations described herein can generally be performed in any order. Furthermore, although various operation flowcharts are shown in one or more sequences, it should be understood that the various operations may be performed in other orders than those shown, or may be performed simultaneously. Unless the context otherwise requires, examples of such alternative orderings may include overlapping, interleaving, interruption, reordering, incremental, preparatory, supplementary, simultaneous, reverse, or other altered orderings. Moreover, unless the context otherwise requires, terms such as “in response to,” “related,” or other past tense adjectives are generally not intended to exclude such variations.

[0386] It is worth noting that any reference to "one aspect," "one aspect," "one example," or "one example" means that the specific feature, structure, or characteristic described in connection with said aspect is included in at least one aspect. Therefore, the phrases "in one aspect," "in one aspect," "in one example," and "in one example" appearing in various places throughout the specification do not necessarily refer to the same aspect. Furthermore, specific features, structures, or characteristics may be combined in one or more aspects in any suitable manner.

[0387] Any patent application, patent, non-patent publication, or other public material mentioned in this specification and / or listed in any application data sheet is incorporated herein by reference, provided that the incorporated material is inconsistent with this specification. Therefore, and to the extent necessary, the disclosures expressly listed herein replace any conflicting material incorporated herein by reference. Any material or portion thereof allegedly incorporated herein by reference that conflicts with existing definitions, statements, or other public materials listed herein will be incorporated only to the extent that the incorporated material does not conflict with existing public materials.

[0388] In summary, many beneficial effects resulting from employing the concepts described herein have been described. For illustrative and descriptive purposes, one or more of the specific embodiments described above have been provided. These embodiments are not intended to be exhaustive or limited to the precise forms disclosed in the invention. Modifications or variations may be made to the invention in accordance with the teachings above. The one or more forms chosen and described are intended to illustrate the principles and practical applications, thereby enabling those skilled in the art to utilize various forms and modifications suitable for the intended particular use. The claims filed herein are intended to define the full scope.

Claims

1. A control system for a surgical system, the control system comprising: an imaging system, the imaging system comprising: an emitter configured to emit multi-spectral electromagnetic radiation (EMR); and an image sensor; and a control circuit coupled to the imaging system, the control circuit configured to: detect airborne particulates via the image sensor based on the emitted EMR; determine a boundary of a particulate cloud containing the detected airborne particulates; determine a characteristic of the boundary of the particulate cloud, wherein the characteristic of the boundary comprises a direction of movement of the boundary; and adjust a control parameter of the surgical system according to the characteristic of the boundary of the particulate cloud.

2. The control system of claim 1, wherein, the control circuit is further configured to: detect a change in the characteristic of the boundary of the particulate cloud; and adjust the control parameter from a first value to a second value according to the change in the characteristic.

3. The control system of claim 1, wherein: the surgical system comprises a generator configured to drive an electrosurgical instrument; and the control parameter comprises a duty cycle of energy of the generator.

4. The control system of claim 1, wherein: the surgical system comprises a smoke evacuator; and the control parameter comprises at least one of a fan state or a motor state of the smoke evacuator.

5. A control system for a surgical system, the control system comprising: an imaging system, the imaging system comprising: a multi-spectral electromagnetic radiation (EMR) source; and an image sensor; and a control circuit coupled to the imaging system, the control circuit configured to: detect EMR reflected from particulates in an aerosol received by the image sensor; determine a configuration of the aerosol containing the detected particulates; and wherein at least one of a first control parameter or a second control parameter of the surgical system is adjusted according to the determined configuration of the aerosol, the configuration comprising a direction of movement of the aerosol.

6. The control system of claim 5, wherein, the control circuit is further configured to: detect a change in the configuration of the aerosol; and adjust at least one of the first control parameter or the second control parameter from a first value to a second value according to the change in the configuration.

7. The control system of claim 5, wherein: the surgical system comprises a generator configured to drive an electrosurgical instrument; and the first control parameter comprises a duty cycle of energy of the generator.

8. The control system of claim 5, wherein: the surgical system comprises a smoke evacuator; and the second control parameter comprises at least one of a fan state or a motor state of the smoke evacuator.

9. A method of controlling a surgical system, the surgical system comprising an imaging system, the imaging system comprising an emitter configured to emit multi-spectral electromagnetic radiation (EMR) and an image sensor; the method comprising: detecting airborne particulates via the image sensor based on the emitted EMR; determining a boundary of a particulate cloud containing the detected airborne particulates; determining a characteristic of a boundary of the plume, wherein the characteristic of the boundary comprises a direction of movement of the boundary; and adjusting a control parameter of the surgical system in accordance with the characteristic of the boundary of the plume.

10. The method of claim 9, wherein, the control circuit is further configured to: detect a change in the characteristic of the boundary of the plume; and adjust the control parameter from a first value to a second value in accordance with the change in the characteristic.

11. The method of claim 9, wherein: the surgical system comprises a generator configured to drive an electrosurgical instrument; and the control parameter comprises an energy duty cycle of the generator.

12. The method of claim 9, wherein: the surgical system comprises a smoke evacuator; and the control parameter comprises at least one of a fan state or a motor state of the smoke evacuator.

Images