Systems and methods for determining, adjusting, and managing resection margins around patient tissue

Abstract

The present disclosure relates to a surgical visualization system that may include a structured light emitter, a spectral light emitter, an image sensor, and control circuitry. The structured light emitter may emit a structured pattern of electromagnetic radiation onto an anatomical structure. The spectral light emitter may emit electromagnetic radiation comprising a plurality of wavelengths. At least one of the wavelengths may penetrate a portion of the anatomical structure and reflect from patient tissue. The image sensor may detect the structured pattern of electromagnetic radiation reflected from the anatomical structure and the at least one wavelength reflected from the patient tissue. The control circuitry may receive a signal from the image sensor and, based on the at least one signal received from the image sensor, construct a model of the anatomical structure, detect a position of the patient tissue, and determine a margin surrounding the patient tissue.

Description

Background Art

[0001] Surgical systems are often combined with imaging systems that allow clinicians to view the surgical site and / or one or more portions thereof, for example, on one or more displays (such as monitors). The displays can be local to the operating room and / or remote. The imaging system may include a scope with a camera that views the surgical site and transmits the view to a display that the clinician can view. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can identify and / or convey to clinicians. For example, some imaging systems may not be able to identify certain hidden structures, physical contours, and / or dimensions in three-dimensional space during surgery. Additionally, some imaging systems may not be able to transmit and / or convey certain information to clinicians during surgery. Summary of the Invention

[0002] In one general aspect, a surgical visualization system is disclosed that includes a structured light emitter configured to emit a structured pattern of electromagnetic radiation onto an anatomical structure, and a spectral light emitter configured to emit electromagnetic radiation comprising a plurality of wavelengths. At least one of the plurality of wavelengths is selected to penetrate a portion of the anatomical structure and reflect from patient tissue. An image sensor is configured to detect the structured pattern of electromagnetic radiation reflected from the anatomical structure and the at least one wavelength reflected from the patient tissue. Control circuitry is configured to receive a first signal and a second signal from the image sensor, construct a model of the anatomical structure based on the first signal, detect a position of the patient tissue relative to the model of the anatomical structure based on the second signal, and determine a margin around the patient tissue based at least in part on the position of the patient tissue.

[0003] In another general aspect, a surgical visualization system is disclosed that includes a light emitter configured to emit electromagnetic radiation comprising a plurality of wavelengths and a plurality of frequencies toward an anatomical structure. A portion of the electromagnetic radiation is configured to reflect from a surface of the anatomical structure, and a portion of the electromagnetic radiation is configured to penetrate the surface of the anatomical structure and reflect from at least one of patient tissue and features of the anatomical structure. An image sensor is configured to detect the portion of the electromagnetic radiation that reflects from the surface of the anatomical structure. The image sensor is further configured to detect the portion of the electromagnetic radiation that penetrates the surface of the anatomical structure and reflects from at least one of patient tissue and features of the anatomical structure. A control circuit is configured to receive a signal from the image sensor and determine an edge of at least one of the patient tissue and features surrounding the anatomical structure based at least in part on the signal. The signal is associated with the reflected electromagnetic radiation.

[0004] In yet another general aspect, a surgical visualization system is disclosed that includes a spectral imaging system configured to emit electromagnetic radiation comprising a plurality of wavelengths and a structured light system configured to emit a structured pattern of electromagnetic radiation onto a surface of an anatomical structure. The spectral imaging system is configured to detect a first structure within the anatomical structure based on reflection of a first wavelength of the plurality of wavelengths. The structured light system is further configured to generate a three-dimensional model of the anatomical structure. A sensor system is configured to detect the reflected electromagnetic radiation. A control system is configured to determine a margin surrounding the first structure within the anatomical structure based on reflection of the first wavelength of the plurality of wavelengths. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The novel features characteristic of the various aspects are set forth with particularity in the appended claims. However, the described aspects, both as to organization and method of operation, may be best understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0006] Figure 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0007] Figure 2 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0008] Figure 2A Control circuitry configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0009] Figure 2B Combinatorial logic circuitry configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure is shown.

[0010] Figure 2C A sequential logic circuit configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0011] Figure 3 is a description according to at least one aspect of the present disclosure Figure 1 Triangulation between surgical devices, imaging devices and critical structures to determine the depth of critical structures below the tissue surface A Schematic diagram of .

[0012] Figure 4 is a schematic diagram of a surgical visualization system configured to identify a critical structure below a tissue surface according to at least one aspect of the present disclosure, wherein the surgical visualization system includes a method for determining a depth d of the critical structure below the tissue surface. A Pulsed light source.

[0013] Figure 5 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0014] Figure 6 is a schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, wherein the surgical visualization system is configured to identify critical structures embedded within tissue.

[0015] Figure 7A and Figure 7B According to at least one aspect of the present disclosure Figure 6 A 3D camera captures a view of the key structure, where Figure 7A is the view from the left lens of the 3D camera, and Figure 7B is the view from the right lens of the 3D camera.

[0016] Figure 8 According to at least one aspect of the present disclosure Figure 6 Schematic diagram of a surgical visualization system in which a camera-key structure distance d from a three-dimensional camera to a key structure can be determined w .

[0017] Figure 9 is a schematic diagram of a surgical visualization system utilizing two cameras to determine the position of an embedded critical structure according to at least one aspect of the present disclosure.

[0018] Figure 10A is a schematic diagram of a surgical visualization system utilizing a camera that is axially moved between a plurality of known positions to determine the position of an embedded critical structure in accordance with at least one aspect of the present disclosure.

[0019] Figure 10B According to at least one aspect of the present disclosure Figure 10A Schematic diagram of a surgical visualization system in which a camera is moved axially and rotationally between multiple known positions to determine the position of embedded critical structures.

[0020] Figure 11 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0021] Figure 12 is a schematic diagram of a structured light source for a surgical visualization system according to at least one aspect of the present disclosure.

[0022] Figure 13A is a graph of absorption coefficients at different wavelengths for various biological materials according to at least one aspect of the present disclosure.

[0023] Figure 13B is a schematic diagram of visualizing an anatomical structure via a spectral surgical visualization system according to at least one aspect of the present disclosure.

[0024] Figures 13C to 13E Depicted are exemplary hyperspectral recognition features for distinguishing anatomical structures from obscurants in accordance with at least one aspect of the present disclosure, wherein Figure 13C is a graphic representation of the ureteral features and obscurations, Figure 13D is a graphical representation of arterial features and obscurations, and Figure 13E is a graphical representation of neural signatures and masking.

[0025] Figure 14 is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distances to critical anatomical structures according to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) positioned on a common device.

[0026] Figure 15 According to at least one aspect of the present disclosure Figure 14 Schematic diagram of the transmitted wave, received wave, and the delay between the transmitted and received waves of the NIR time-of-flight measurement system.

[0027] Figure 16An NIR time-of-flight measurement system configured to sense distances to different structures, including a transmitter (emitter) and a receiver (sensor) on separate devices, is shown in accordance with at least one aspect of the present disclosure.

[0028] Figure 17 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0029] Figure 18 A surgical system for performing a surgical procedure in an operating room according to at least one aspect of the present disclosure.

[0030] Figure 19 A computer-implemented interactive surgical system according to at least one aspect of the present disclosure is shown.

[0031] Figure 20 A diagram illustrating a situational awareness surgical system in accordance with at least one aspect of the present disclosure is shown.

[0032] Figure 21 Shown is a timeline depicting a hub's situational awareness in accordance with at least one aspect of the present disclosure.

[0033] Figure 22 A display of a surgical visualization system is shown according to at least one aspect of the present disclosure.

[0034] Figure 23A A model of an anatomical structure generated by a surgical visualization system is shown according to at least one aspect of the present disclosure.

[0035] Figure 23B According to at least one aspect of the present disclosure Figure 23A The model of the display.

[0036] Figure 24A Another model of an anatomical structure generated by a surgical visualization system is shown in accordance with at least one aspect of the present disclosure.

[0037] Figure 24B According to at least one aspect of the present disclosure Figure 24A The model of the display.

[0038] Figure 25 Another model of an anatomical structure generated by a surgical visualization system is shown in accordance with at least one aspect of the present disclosure.

[0039] Figure 26 According to at least one aspect of the present disclosure Figure 25 The model of the display.

[0040] Figure 27FIG. 4 is a display of another model of an anatomical structure generated by a surgical visualization system according to at least one aspect of the present disclosure. DETAILED DESCRIPTION

[0041] The applicant of the present application owns the following U.S. patent applications, each of which is incorporated herein by reference in its entirety:

[0042] Attorney Docket No. END9228USNP1 / 190580-1M, entitled “METHOD OF USING IMAGING DEVICES IN SURGERY”;

[0043] Attorney Docket No. END9227USNP1 / 190579-1, entitled “ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM”;

[0044] Attorney Docket No. END9226USNP1 / 190578-1, titled “SURGICAL SYSTEM CONTROLBASED ON MULTIPLE SENSED PARAMETERS”;

[0045] Attorney Docket No. END9225USNP1 / 190577-1, entitled “ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS”;

[0046] Attorney Docket No. END9224USNP1 / 190576-1, titled “ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS”;

[0047] Attorney Docket No. END9223USNP1 / 190575-1, entitled “SURGICAL SYSTEMSCORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA”;

[0048] Attorney Docket No. END9222USNP1 / 190574-1, entitled “SURGICAL SYSTEMS FORGENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED”;

[0049] Attorney Docket No. END9221USNP1 / 190573-1, entitled “SURGICAL SYSTEM FOROVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN”;

[0050] Attorney Docket No. END9220USNP1 / 190572-1, entitled “SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS”;

[0051] · Attorney Docket No. END9218USNP1 / 190570-1, whose name is

[0052] "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT";

[0053] Attorney Docket No. END9217USNP1 / 190569-1, entitled “DYNAMIC SURGICALVISUALIZATION SYSTEMS”; and

[0054] ·Agent Docket No. END9216USNP1 / 190568-1, whose name is

[0055] "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM".

[0056] The applicant of the present application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety:

[0057] U.S. patent application serial number 16 / 354,417, entitled “Input Controls for Robotics Urgery”;

[0058] U.S. patent application serial number 16 / 354,420, entitled “Dual Mode Controls for Robotic Surgery”;

[0059] U.S. patent application serial number 16 / 354,422, entitled “MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY”;

[0060] U.S. patent application serial number 16 / 354,440, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY”

[0061] U.S. patent application serial number 16 / 354,444, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE”

[0062] U.S. patent application serial number 16 / 354,454, entitled “ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS”

[0063] U.S. patent application Ser. No. 16 / 354,461, entitled “SELECTABLE VARIABLE RESPONSEOF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS”

[0064] U.S. patent application serial number 16 / 354,470, entitled “SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS”

[0065] U.S. patent application serial number 16 / 354,474, entitled “ROBOTIC SURGICAL CONTROLSHAVING FEEDBACK CAPABILITIES”

[0066] U.S. patent application Ser. No. 16 / 354,478, entitled “ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK”; and

[0067] U.S. patent application serial number 16 / 354,481, entitled “JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS.”

[0068] The applicant of the present application also owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety:

[0069] U.S. patent application serial number 16 / 128,179, entitled “SURGICAL VISUALIZATION PLATFORM”;

[0070] U.S. patent application serial number 16 / 128,180, entitled “CONTROLLING AN EMITTERASSEMBLY PULSE SEQUENCE”;

[0071] U.S. patent application serial number 16 / 128,198, entitled “SINGULAR EMR SOURCE EMITTER ASSEMBLY”;

[0072] U.S. patent application serial number 16 / 128,207, entitled “COMBINATION EMITTER AND CAMERA ASSEMBLY”;

[0073] U.S. patent application serial number 16 / 128,176, entitled “SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES”;

[0074] U.S. patent application serial number 16 / 128,187, entitled “Surgical Visalization of Multiple Targets”;

[0075] U.S. patent application serial number 16 / 128,192, entitled “VISUALIZATION OF SURGICALDEVICES”;

[0076] U.S. patent application serial number 16 / 128,163, entitled “OPERATIVE COMMUNICATION OFLIGHT”;

[0077] U.S. patent application serial number 16 / 128,197, entitled “ROBOTIC LIGHT PROJECTION TOOLS”;

[0078] U.S. patent application serial number 16 / 128,164, entitled “SURGICAL VISUALIZATION FEEDBACK SYSTEM”;

[0079] U.S. patent application serial number 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MOITORING”;

[0080] U.S. patent application serial number 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA”;

[0081] U.S. Patent Application Serial No. 16 / 128,170, entitled “ROBOTICALLY-

[0082] ASSISTED SURGICAL SUTURING SYSTEMS”;

[0083] U.S. patent application serial number 16 / 128,183, entitled “SAFETY LOGIC FOR SURGICALSUTURING SYSTEMS”;

[0084] U.S. patent application serial number 16 / 128,172, entitled “ROBOTIC SYSTEM WITH SEPARATEPHOTOACOUSTIC RECEIVER”; and

[0085] U.S. patent application serial number 16 / 128,185, entitled “FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION.”

[0086] The applicant of the present application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0087] U.S. patent application serial number 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201111;

[0088] U.S. patent application serial number 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201142;

[0089] U.S. patent application Ser. No. 15 / 940,711, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201120; and

[0090] U.S. patent application serial number 15 / 940,722, entitled

[0091] “CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY,” now U.S. Patent Application Publication No. 2019 / 0200905.

[0092] The applicant of this patent application owns the following U.S. patent applications filed on December 4, 2018, the disclosures of each of these provisional patent applications are incorporated herein by reference in their entirety:

[0093] U.S. patent application serial number 16 / 209,395, entitled “METHOD OF HUB COMMUNICATION,” now U.S. Patent Application Publication No. 2019 / 0201136;

[0094] U.S. patent application serial number 16 / 209,403, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” now U.S. Patent Application Publication No. 2019 / 0206569;

[0095] U.S. patent application serial number 16 / 209,407, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL,” now U.S. Patent Application Publication No. 2019 / 0201137;

[0096] U.S. patent application serial number 16 / 209,416, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS,” now U.S. Patent Application Publication No. 2019 / 0206562;

[0097] U.S. patent application Ser. No. 16 / 209,423, entitled “METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS,” now U.S. Patent Application Publication No. 2019 / 0200981;

[0098] U.S. patent application Ser. No. 16 / 209,427, entitled “METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIOFREQUENCY DEVICES,” now U.S. Patent Application Publication No. 2019 / 0208641;

[0099] U.S. patent application Ser. No. 16 / 209,433, entitled “METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB,” now U.S. Patent Application Publication No. 2019 / 0201594;

[0100] U.S. patent application serial number 16 / 209,447, entitled “METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB,” now U.S. Patent Application Publication No. 2019 / 0201045;

[0101] U.S. patent application serial number 16 / 209,453, entitled “METHOD FOR CONTROLLING SMARTENERGY DEVICES,” now U.S. Patent Application Publication No. 2019 / 0201046;

[0102] U.S. patent application serial number 16 / 209,458, entitled “METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE,” now U.S. Patent Application Publication No. 2019 / 0201047;

[0103] U.S. patent application Ser. No. 16 / 209,465, entitled “METHOD FOR ADAPTIVE CONTROLSCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION,” now U.S. Patent Application Publication No. 2019 / 0206563;

[0104] U.S. patent application Ser. No. 16 / 209,478, entitled “METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE,” now U.S. Patent Application Publication No. 2019 / 0104919;

[0105] U.S. patent application Ser. No. 16 / 209,490, entitled “METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION,” now U.S. Patent Application Publication No. 2019 / 0206564; and

[0106] U.S. patent application serial number 16 / 209,491, entitled “METHOD FOR CIRCULAR STAPLERCONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS,” now U.S. Patent Application Publication No. 2019 / 0200998.

[0107] Before describing in detail various aspects of the surgical visualization platform, it should be noted that the illustrative examples are not limited in application or use to the details of construction and arrangement of components shown in the drawings and the specification. The illustrative examples may be implemented or incorporated in other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise indicated, the terms and expressions used herein are selected for the convenience of the reader and are not intended to be limiting. Furthermore, it should be understood that one or more of the aspects, expressions of aspects, and / or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and / or examples described below.

[0108] Surgical visualization system

[0109] The present disclosure relates to a surgical visualization platform that utilizes "digital surgery" to obtain additional information about a patient's anatomy and / or surgical procedure. The surgical visualization platform is further configured to communicate the data and / or information to one or more clinicians in a helpful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomy and / or surgical procedure.

[0110] "Digital surgery" can encompass robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and more. Although the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In some cases, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0111] In some cases, a surgical system incorporating a surgical visualization platform can implement intelligent dissection to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve and / or tumors. In other cases, the critical structure can be, for example, an alien structure in the dissection field, such as a surgical device, surgical fastener, clamp, tack, bougie, band and / or plate. The critical structure can be determined based on different patients and / or different surgeries. Exemplary critical structures are also described herein. For example, intelligent dissection technology can provide improved intraoperative guidance for dissection and / or key anatomical structure detection and avoidance technology can be used to implement intelligent decision-making.

[0112] Surgical systems incorporating surgical visualization platforms can also implement intelligent anastomosis technology that utilizes improved workflows to provide more consistent anastomosis at optimal locations. Various surgical visualization platforms and procedures described herein can also be utilized to improve cancer localization technology. For example, cancer localization technology can identify and track cancer location, orientation, and margins. In some cases, cancer localization technology can compensate for movement of tools, patients, and / or patient anatomy during surgery to provide guidance to the clinician returning to a point of interest.

[0113] In certain aspects of the present disclosure, a surgical visualization platform can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly when dissecting and / or placing suturing devices within tissue. Certain tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0114] During surgery, the information available to the clinician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures (such as structures embedded or buried within an organ) may be at least partially concealed or hidden from view. Additionally, certain sizes and / or relative distances may be difficult to detect using existing sensor systems and / or difficult to perceive with the "naked eye." Furthermore, certain structures may move preoperatively (e.g., before surgery but after a preoperative scan) and / or intraoperatively. In such cases, the clinician may not be able to accurately determine the location of critical structures intraoperatively.

[0115] When the orientation of a critical structure is uncertain and / or when the proximity between a critical structure and a surgical tool is unknown, the clinician's decision-making process may be hindered. For example, a clinician may avoid certain areas in order to avoid accidentally dissecting a critical structure; however, the avoided area may be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or excessive / overly cautious operation, the clinician may be unable to enter certain desired areas. For example, excessive caution may cause the clinician to leave behind a portion of a tumor and / or other undesirable tissue in an attempt to avoid a critical structure, even if the critical structure is not in that particular area and / or will not be negatively affected by the clinician working in that particular area. In some cases, surgical results can be improved by increasing knowledge and / or certainty, which can make the surgeon more accurate in specific anatomical areas and, in some cases, make the surgeon less conservative / more aggressive.

[0116] In various aspects, the present disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities that go beyond what a clinician can see with the "naked eye" and / or beyond what an imaging system can identify and / or convey to the clinician. Various surgical visualization systems can enhance and strengthen what a clinician can know before tissue treatment (e.g., dissection) and, therefore, can improve outcomes in various situations.

[0117] For example, a visualization system may include a first light emitter configured to emit multiple spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to the spectral waves (spectral imaging), and / or the light pattern. It should be noted that throughout the following disclosure, unless specifically mentioned as visible light, any reference to "light" may include electromagnetic radiation (EMR) or photons in the visible and / or invisible portions of the EMR wavelength spectrum. The surgical visualization system may also include an imaging system and control circuitry in signal communication with the receiver and the imaging system. Based on the output from the receiver, the control circuitry may determine a geometric surface map (i.e., a three-dimensional surface topography) of a visible surface at the surgical site and one or more distances relative to the surgical site. In some cases, the control circuitry may determine one or more distances to at least partially concealed structures. Furthermore, the imaging system may communicate the geometric surface map and the one or more distances to the clinician. In such cases, the enhanced view of the surgical site provided to the clinician may provide a representation of concealed structures within the surroundings of the surgical site. For example, the imaging system can virtually enhance the hidden structure on a geometric surface map of the hidden and / or obstructing tissue, similar to a line drawn on the ground to indicate a practical line below the surface. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to visible obstructing tissue and / or to at least partially obstructed structures and / or the depth of the hidden structure below the visible surface of the obstructing tissue. For example, the visualization system can determine the distance of the enhancement line relative to the surface of the visible tissue and communicate the distance to the imaging system.

[0118] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide valuable information to a clinician during a surgical procedure. Thus, for example, the clinician knows that the surgical visualization system is tracking critical structures that can be approached during dissection (such as ureters, specific nerves and / or critical blood vessels), and can confidently maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide instructions to the clinician for a sufficient period of time to enable the clinician to pause and / or slow down the surgical procedure and assess the proximity to the critical structure to prevent accidental damage thereto. The surgical visualization system can provide the clinician with an ideal, optimized and / or customizable amount of information to allow the clinician to confidently and / or quickly move through tissue while avoiding accidental damage to healthy tissue and / or critical structures, and thereby minimizing the risk of injury caused by the surgical procedure.

[0119] Figure 1 is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In some cases, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured to identify critical structures intraoperatively and / or facilitate surgical device avoidance of critical structures 101. For example, by identifying critical structures 101, a clinician can avoid maneuvering a surgical device around a critical structure 101 and / or an area within a predetermined proximity of a critical structure 101 during a surgical procedure. For example, a clinician can avoid dissecting veins, arteries, nerves, and / or blood vessels identified as critical structures 101 and / or avoid dissecting near such critical structures. In various cases, the critical structures 101 can be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0120] The surgical visualization system 100 incorporates tissue recognition and geometric surface mapping in conjunction with the distance sensor system 104. Combined, these features of the surgical visualization system 100 can determine the position of critical structures 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system comprising, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various embodiments, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Views from the imaging device 120 can be provided to the clinician, and in various aspects of the present disclosure, these views can be enhanced with additional information based on tissue recognition, topographic mapping, and the distance sensor system 104. In such cases, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem, which can cooperate to provide clinicians with advanced data synthesis and integrated information during surgery.

[0121] The imaging device may include a camera or imaging sensor configured to detect, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). In various aspects of the present disclosure, the imaging system may include, for example, an imaging device, such as an endoscope. Additionally or alternatively, the imaging system may include, for example, an imaging device, such as an arthroscope, an angioscope, a bronchoscope, a choledochoscope, a colonoscope, a cystoscope, a duodenoscope, an enteroscope, an esophagogastroduodenoscope (gastroscope), a laryngoscope, a nasopharyngeal-nephroscope, a sigmoidoscope, a thoracoscope, a ureteroscope, or an exoscope. In other cases, such as in open surgical applications, the imaging system may not include a viewing scope.

[0122] In various aspects of the present disclosure, the tissue identification subsystem may be implemented using a spectral imaging system. The spectral imaging system may rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” issued on March 1, 2016, which is incorporated herein by reference in its entirety.

[0123] In various aspects of the present disclosure, the surface mapping subsystem can be implemented using a light patterning system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be used in the surgical visualization system described herein.

[0124] Structured light is the process of projecting a known pattern (usually a grid or horizontal stripes) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, are incorporated herein by reference in their entirety.

[0125] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and to determine various distances relative to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to tissue (or other structures) identified at a surgical site.

[0126] Figure 2 is a schematic diagram of a control system 133 that may be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 in signal communication with a memory 134. The memory 134 stores instructions executable by the control circuit 132 to determine and / or identify critical structures (e.g., Figure 1 101 ), determine and / or calculate one or more distances and / or three-dimensional digital representations, and transmit certain information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (e.g., Figure 1The imaging device 120 in FIG. 1 ), one or more displays 146, or one or more controls 148, or any combination thereof. The camera 144 may include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, 3D lenses, etc.) that emit light in various visible and non-visible light spectrums. The display 146 may include one or more screens or monitors for depicting real, virtual, and / or virtual-augmented images and / or information to one or more clinicians.

[0127] In various aspects, the heart of camera 144 is the image sensor 135. Generally speaking, modern image sensors 135 are solid-state electronic devices containing up to millions of discrete photodetector sites (called pixels). Image sensor 135 technology falls into one of two categories: charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 are sensitive to wavelengths from approximately 350 nm to 1050 nm, but this range is often given as 400 nm to 1000 nm. Generally speaking, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, therefore, cannot distinguish colors. Consequently, there are two types of color CCD cameras: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution and use a mosaic (e.g., Bayer) optical filter to separate the incoming light into a series of colors and employ an interpolation algorithm to resolve the full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing a prism to direct each portion of the incoming light spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value, rather than using an algorithm to determine color. Three-chip cameras offer extremely high resolution.

[0128] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, the single light source can be pulsed to provide light in the invisible spectrum (e.g., infrared spectrum light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic component 140 can identify key structures via data from the spectral light source 150 received in part by the image sensor 135 of the camera 144. The surface mapping logic component 136 can determine the surface contour of the visible tissue based on the reflected structured light. Using the time of flight measurement results, the distance determination logic component 141 can determine one or more distances to the visible tissue and / or key structure 101. One or more outputs from the surface mapping logic 136 , tissue identification logic 140 , and distance determination logic 141 may be provided to the imaging logic 138 and may be combined, blended, and / or overlaid for communication to the clinician via a display 146 of the imaging system 142 .

[0129] The manual now briefly turns to Figures 2A to 2C , to describe various aspects of the control circuitry 132 for controlling various aspects of the surgical visualization system 100. Figure 2A , shows a control circuit 400 configured to control various aspects of the surgical visualization system 100 according to at least one aspect of the present disclosure. The control circuit 400 can be configured to implement the various processes described herein. The control circuit 400 can include a microcontroller comprising one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions to implement the various processes described herein. The processor 402 can be any of a variety of single-core or multi-core processors known in the art. The memory circuit 404 can include volatile storage media and non-volatile storage media. The processor 402 can include an instruction processing unit 406 and an operation unit 408. The instruction processing unit can be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0130] Figure 2B1. Combinatorial logic circuit 410 is shown that is configured to control various aspects of surgical visualization system 100 in accordance with at least one aspect of the present disclosure. Combinatorial logic circuit 410 can be configured to implement various processes described herein. Combinatorial logic circuit 410 can include a finite state machine that includes a combinatorial logic component 412 that is configured to receive data associated with a surgical instrument or tool at an input 414, process the data via combinatorial logic component 412, and provide an output 416.

[0131] Figure 2C A sequential logic circuit 420 is shown that is configured to control various aspects of the surgical visualization system 100 in accordance with at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic component 422 may be configured to implement the various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, a combinational logic component 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 may store a current state of the finite state machine. In some cases, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic component 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinational logic component 422, and provide an output 428. In other aspects, the circuit may include a processor (e.g., Figure 2A In other aspects, the finite state machine may include a combinational logic circuit (e.g., combinational logic circuit 410, Figure 2B ) and a combination of a sequential logic circuit 420.

[0132] See again Figure 1 In the surgical visualization system 100 of the present invention, the key structure 101 can be an anatomical structure of interest. For example, the key structure 101 can be an anatomical structure such as a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor. In other cases, the key structure 101 can be, for example, a foreign structure in the anatomical field, such as a surgical device, a surgical fastener, a clamp, a tack, a bougie, a band, and / or a plate. Exemplary key structures are further described herein and in the aforementioned concurrently filed U.S. patent applications (including, for example, U.S. patent application Ser. No. 16 / 128,192, filed on September 11, 2018, entitled “VISUALIZATION OF SURGICAL DEVICES”), which are incorporated herein by reference in their entirety.

[0133] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be positioned below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the critical structure 101 from the clinician's view. From the perspective of the imaging device 120, the critical structure 101 is also obscured by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other cases, the critical structure 101 can be partially obscured from view.

[0134] Figure 1 Also depicted is a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from a distal end of a shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device, such as, for example, a dissector, a stapler, a grasper, a clip applier, and / or an energy device (including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector). Additionally or alternatively, the surgical device 102 can include, for example, another imaging or diagnostic modality, such as an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to enable identification of one or more critical structures 101 and the proximity of the surgical device 102 to the critical structure 101.

[0135] The imaging device 120 of the surgical visualization system 100 is configured to be capable of detecting various wavelengths of light, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (such as a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens that are used together to simultaneously record two two-dimensional images, and thereby generate a three-dimensional image of the surgical site, render a three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to be capable of receiving images indicating the topography of visible tissue and the identification and orientation of hidden critical structures, as further described herein. For example, the field of view of the imaging device 120 may overlap with the pattern of light (structured light) on the surface 105 of the tissue, such as Figure 1 shown.

[0136] In one aspect, the surgical visualization system 100 can be incorporated into a robotic system 110. For example, the robotic system 110 can include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118, which can include servo motor controls. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. A robotic control unit can be configured to issue control motions to the robotic arms 112, 114, which can affect, for example, the surgical device 102 and the imaging device 120.

[0137] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as stripes, grid lines, and / or dots, to enable determination of the topography or terrain of the surface 105. For example, a projected light array 130 can be used for three-dimensional scanning and registration of the surface 105. The projected light array 130 can be emitted from the emitter 106 located, for example, on the surgical device 102 and / or one of the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the shape defined by the surface 105 of the tissue 103 and / or the intraoperative motion of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances relative to the surface 105.

[0138] In one aspect, the imaging device 120 may further include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable identification of the type of anatomical and / or physical structure (such as the critical structure 101). Identification of the critical structure 101 may be achieved, for example, through spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a photoacoustic imaging system. In other cases, the optical waveform emitter 123 may be located on a surgical device separate from the imaging device 120 .

[0139] The surgical visualization system 100 may also include a distance sensor system 104 that is configured to determine one or more distances at the surgical site. In one aspect, the time of flight distance sensor system 104 can be a time of flight distance sensor system that includes an emitter, such as emitter 106, and a receiver 108 that can be positioned on the surgical device 102. In other cases, the time of flight emitter can be separate from the structured light emitter. In one general aspect, the emitter 106 portion of the time of flight distance sensor system 104 can include a very tiny laser source, and the receiver 108 portion of the time of flight distance sensor system 104 can include a matching sensor. The time of flight distance sensor system 104 can detect the "time of flight" or the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. The use of a very narrow light source in the emitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104. Still referring to Figure 1 , d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 can be used to determine the transmitter-tissue distance d e Device-tissue distance d t The device-tissue distance d can be obtained based on the known position of the transmitter 106 on the axis of the surgical device 102 relative to the distal end of the surgical device 102. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t The transmitter-tissue distance d e In some cases, the shaft of surgical device 102 may include one or more articulation joints and may be capable of articulation relative to transmitter 106 and jaws. The articulation configuration may include, for example, a multi-jointed vertebra-like structure. In some cases, a three-dimensional camera may be used to triangulate one or more distances to surface 105.

[0140] In various instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate surgical device rather than on the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In other instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate robotically controlled arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In some instances, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance from the transmitter 106 on the surgical device 102 to the surface 105 of the tissue 103 using a line between the transmitter 106 on the surgical device 102 and the imaging device 120. For example, the distance d may be determined based on the known positions of the transmitter 106 (on the surgical device 102) and the receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. e The three-dimensional position of the receiver 108 may be known and / or registered intraoperatively with the robot coordinate plane.

[0141] In some cases, the position of the transmitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other cases, the surgical visualization system 100 can be used separately from the robotic system. In such cases, the distance sensor system 104 can be independent of the robotic system.

[0142] In some cases, one or more of the robotic arms 112, 114 may be separate from the main robotic system used during the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and registered to a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or register the position of the robotic arms 112, 114 relative to a specific coordinate system. Similarly, the position of the surgical device 102 and the imaging device 120 may be registered relative to a specific coordinate system.

[0143] Still see Figure 1 , d w is the camera-to-key structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the key structure 101, and d Ais the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d w The use of spectral imaging in conjunction with a time-of-flight sensor is further described herein. Furthermore, see now Figure 3 In various aspects of the present disclosure, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 is A It can be determined by the following method: According to the distance d w and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and therefore the known distance d between them). x ) to perform triangulation to determine the distance d y (It is the distance d e and d A sum).

[0144] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, the first waveform (or range of waveforms) can be used to determine the camera-critical structure distance d w , and the second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0145] Additionally or alternatively, in some cases, the distance d A It can be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance d A This can be determined using spectral imaging because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first manner or amount, and collagen can reduce the detection signal in a different second manner or amount.

[0146] Now see Figure 4 The surgical visualization system 160 in FIG. 1 includes a surgical device 162 including an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform. The optical waveform emitter 123 can be configured to emit a waveform for determining a distance d from a common device such as the surgical device 162. t and d w, as further described herein. In such cases, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is A It can be determined as follows:

[0147] d A =d w -d t .

[0148] As disclosed herein, various information about visible tissue, embedded critical structures, and surgical devices can be determined by utilizing a combined approach that combines an image sensor configured to detect spectral wavelengths and structured light arrays with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Furthermore, the image sensor can be configured to receive visible light and, thus, provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various embodiments, the imaging device 120 may include multiple image sensors.

[0149] Camera-key structure distance d w Detection can also be performed in one or more alternative ways. In one aspect, the key structure 201 can be illuminated using, for example, fluoroscopic visualization techniques such as fluorescent indocyanine green (ICG), such as Figures 6 to 8 The camera 220 may include two optical waveform sensors 222 and 224, which simultaneously capture the left image and the right image of the key structure 201 ( Figure 7A and Figure 7B ). In such cases, the camera 220 may depict the glow of the critical structure 201 below the surface 205 of the tissue 203 and the distance d w The distance between sensors 222 and 224 can be determined based on the known distance between them. In some cases, the distance can be more accurately determined by utilizing more than one camera or by moving the camera between multiple locations. In some aspects, one camera can be controlled by a first robotic arm, and the second camera can be controlled by another robotic arm. In such a robotic system, one camera can be, for example, a slave camera on a slave arm. The slave arm and its camera can be programmed to track the other camera and maintain, for example, a specific distance and / or lens angle.

[0150] In other aspects, the surgical visualization system 100 may employ two separate waveform receivers (ie, cameras / image sensors) to determine d w Now see Figure 9If a critical structure 301 or its contents (eg, a vessel or vessel contents) can emit a signal 302, such as using fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0151] On the other hand, see now Figure 10A and Figure 10B The surgical visualization system may use shaking or moving the camera 440 to determine the distance d w The camera 440 is robotically controlled so that the three-dimensional coordinates of the camera 440 at different positions are known. In various cases, the camera 440 can be pivoted at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., a blood vessel or blood vessel contents) can emit a signal, such as using fluoroscopy, the actual position can be triangulated based on the camera 440 being rapidly moved between two or more known positions. Figure 10A , camera 440 is moved axially along axis A. More specifically, camera 440 is translated a distance d1 along axis A closer to key structure 401 to a position indicated as position 440', such as by moving in and out on a robotic arm. As camera 440 moves distance d1 and the size of the view changes relative to key structure 401, the distance to key structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) can correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, camera 440 can be rotated or swept along an arc between different orientations. Referring now to Figure 10B , the camera 440 moves axially along axis A and rotates around axis A by an angle θ3. The pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. Figure 10B In FIG. 4 , the camera 440 is translated and rotated to position 440 . When the camera 440 is moved and the edge of the view changes with respect to the key structure 401 , the distance to the key structure 401 can be calculated. Figure 10B , the distance d2 may be, for example, 9.01 mm, and the angle θ3 may be, for example, 0.9 degrees.

[0152] Figure 5A surgical visualization system 500 is depicted, which is similar in many respects to surgical visualization system 100. In various embodiments, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light at multiple wavelengths to obtain, for example, spectral images of hidden structures. In various embodiments, imaging device 520 can also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown being used intraoperatively to identify and facilitate avoidance of certain critical structures that are not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (in this example, the uterus).

[0153] The surgical visualization system 500 is configured to be able to determine an emitter-tissue distance d from an emitter 506 on a surgical device 502 to a surface 505 of a uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to detect the presence of a transducer based on the transmitter-tissue distance d e To extrapolate the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 t The surgical visualization system 500 is further configured to determine the tissue-ureter distance d from the ureter 501a to the surface 505. A and the camera-ureter distance d from the imaging device 520 to the ureter 501a w As this article about Figure 1 As described above, for example, the surgical visualization system 500 can utilize, for example, spectral imaging and a time-of-flight sensor to determine the distance d w In various circumstances, the surgical visualization system 500 can determine (eg, triangulate) the tissue-ureter distance d based on other distance and / or surface mapping logic described herein. A (or depth).

[0154] Now see Figure 11 , which depicts a schematic diagram of a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For example, control system 600 is a conversion system that integrates spectral signature tissue recognition and structured light tissue localization to identify key structures, particularly when these structures are obscured by other tissues (such as fat, connective tissue, blood, and / or other organs). Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue from healthy tissue within an organ.

[0155] The control system 600 is configured to implement a hyperspectral imaging and visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field of view. The control system 600 includes conversion logic 648 to convert tissue data into information usable by the surgeon. For example, variable reflectivity based on wavelength relative to an obscuring material can be used to identify critical structures within the anatomy. Furthermore, the control system 600 combines the identified spectral signatures with structured light data within an image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical applications in a system with enhanced image overlays. The technology can be used both intraoperatively and preoperatively with the additional visual information. In various scenarios, the control system 600 is configured to provide a warning to the clinician when approaching one or more critical structures. Various algorithms can be employed to guide robotic automation and semi-automated approaches based on the surgical procedure and proximity to critical structures.

[0156] Projected light arrays are used to determine tissue shape and motion intraoperatively. Alternatively, flash lidar can be used for surface mapping of tissue.

[0157] The control system 600 is configured to detect critical structures and provide image overlays of the critical structures, and to measure distances to the surface of visible tissue and distances to embedded / buried critical structures. In other cases, the control system 600 may measure distances to the surface of visible tissue or detect critical structures and provide image overlays of the critical structures.

[0158] The control system 600 includes a spectrum control circuit 602. For example, the spectrum control circuit 602 may be a field programmable gate array (FPGA) or a Figures 2A to 2C Another suitable circuit configuration is described. Spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. For example, processor 604 can be configured for hyperspectral processing and can utilize C / C++ code. For example, video input processor 606 receives video input including control (metadata) data, such as shutter time, wavelength, and sensor analysis. Processor 604 is configured to process the video input signal from video input processor 606 and provide a video output signal to video output processor 608, which includes, for example, a hyperspectral video output that interfaces with control (metadata) data. Video output processor 608 provides the video output signal to image overlay controller 610.

[0159] The video input processor 606 is coupled to a camera 612 at the patient side via a patient isolation circuit 614. As previously described, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers to isolate the patient from other circuits in the system. The camera 612 receives intraoperative images through optics 632 and an image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include, for example, a CMOS image sensor. Figure 2 Any image sensor technology described herein. In one aspect, camera 612 outputs images at a 14-bit / pixel signal. It should be understood that higher or lower pixel resolutions may be employed without departing from the scope of this disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fused output signal is provided to a video input processor 606 and a laser pulse control circuit 622.

[0160] Laser pulse control circuit 622 controls laser engine 624. Laser engine 624 outputs light at multiple wavelengths (λ1, λ2, λ3, ..., λn), including near-infrared (NIR). Laser engine 624 can operate in multiple modes. In one aspect, laser engine 624 can operate in two modes, for example. In a first mode (e.g., normal operating mode), laser engine 624 outputs an illumination signal. In a second mode (e.g., recognition mode), laser engine 624 outputs RGBG and NIR light. In various cases, laser engine 624 can operate in a polarization mode.

[0161] Light output 626 from the laser engine 624 illuminates the target anatomical structure in the intraoperative surgical site 627. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630, which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) onto the surgical tissue or organ at the surgical site 627. The camera 612 receives the patterned light and the reflected light output by the camera optics 632. The image sensor 634 converts the received light into a digital signal.

[0162] The color RGB fusion circuit 616 also outputs a signal to the image overlay controller 610 and the video input module 636 for reading the laser pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0163] The first video output signal 640 and the second video output signal 642 include data representing the location of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. In combination with the data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine the distance d to the buried critical structure. A ( Figure 1 ) (e.g., via triangulation algorithm 644), and the distance d A The image overlay controller 610 may be provided via the video output processor 646. The conversion logic components described above may include the conversion logic circuit 648, the intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0164] In various cases, preoperative data 650 from a CT or MRI scan may be used to register or match certain three-dimensional deformable tissues. Such preoperative data 650 may be provided to an integration module 643 and ultimately to an image overlay controller 610 so that such information may be overlaid with the view from a camera 612 and provided to a video monitor 652. Registration of preoperative data is further described herein and in the aforementioned concurrently filed U.S. patent applications (including, for example, U.S. patent application Ser. No. 16 / 128,195, filed Sep. 11, 2018, entitled "INTEGRATION OF IMAGING DATA"), which are incorporated herein by reference in their entirety.

[0165] The video monitor 652 can output the integrated / enhanced view from the image overlay controller 610. The clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view in which a three-dimensional rendering of visible tissue is depicted and (B) an enhanced view in which one or more hidden critical structures are depicted on the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can, for example, switch distance measurements to the surface of one or more hidden critical structures and / or visible tissue.

[0166] The control system 600 and / or its various control circuits may be incorporated into the various surgical visualization systems disclosed herein.

[0167] Figure 12 7. A structured (or patterned) light system 700 is shown in accordance with at least one aspect of the present disclosure. As described herein, structured light in the form of stripes or lines, for example, can be projected from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. Figure 1 ) can be configured to detect a projected pattern of light on surface 705. The way in which the projected pattern deforms when it strikes surface 705 allows the vision system to calculate depth and surface information of the target anatomy.

[0168] In some cases, invisible (or imperceptible) structured light can be used without interfering with other computer vision tasks where the projected pattern might confuse the image. For example, infrared light that alternates between two diametrically opposed patterns or extremely fast visible light frame rates can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0169] As described above, the various surgical visualization systems described herein can be used to visualize various different types of tissue and / or anatomical structures, including tissue and / or anatomical structures that may be obscured from visualization by EMR in the visible portion of the spectrum. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissue based on different combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the tissue's absorption coefficient at various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on a particular combination of constituent materials. To illustrate, Figure 13A 23 is a graph 2300 depicting how the absorption coefficient of various biological materials varies across the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient of the biological material (e.g., in cm -1), and horizontal axis 2304 represents EMR wavelength (e.g., in μm). Graph 2300 further illustrates a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxyhemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxyhemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials, and thus the tissue types visualized by the surgical visualization system can be identified and distinguished based on the specific combination of constituent materials detected. Accordingly, the spectral imaging system can be configured to emit EMR at multiple different wavelengths, determine the constituent materials of the tissue based on the absorbed EMR responses detected at the different wavelengths, and then characterize the tissue type based on the specific detected combination of constituent materials.

[0170] Figure 13B The use of spectral imaging techniques to visualize different tissue types and / or anatomical structures is shown. Figure 13B In FIG. 2 , a spectral emitter 2320 (e.g., spectral light source 150) is used by the imaging system to visualize a surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from tissue and / or structures at the surgical site 2325 can be imaged by the image sensor 135 ( Figure 2 ) to visualize tissue and / or structures; the tissue and / or structure may be visible (e.g., located on the surface of the surgical site 2325) or obscured (e.g., located beneath other tissue and / or structures at the surgical site 2325). In this example, the imaging system 142 ( Figure 2 ) can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissue that does not conform to a known or expected spectral signature) based on spectral signatures characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials of each of the different tissue / structure types. The visualized tissues and structures can be displayed on a display associated with or coupled to the imaging system 142, such as the imaging system display 146 ( Figure 2 )、Main display 2119( Figure 18 ), non-sterile display 2109 ( Figure 18 ), hub display 2215( Figure 19 )、Device / Equipment Display 2237( Figure 19 )wait.

[0171] Furthermore, the imaging system 142 can be configured to customize or update the displayed visualization of the surgical site based on the identified tissue and / or structure type. For example, the imaging system 142 can display a margin 2330a associated with the tumor 2332 being visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue that should be removed to ensure complete removal of the tumor 2332. The control system 133( Figure 2 ) can be configured to control or update the size of edge 2330a based on tissue and / or structure identified by imaging system 142. In the illustrated example, imaging system 142 has identified multiple anomalies 2338 within the FOV. Accordingly, control system 133 can adjust displayed edge 2330a to a first updated edge 2330b, which is of sufficient size to encompass anomalies 2338. Additionally, imaging system 142 has identified an artery 2334 that partially overlaps with initially displayed edge 2330a (as indicated by highlighted region 2336 of artery 2334). Accordingly, control system 133 can adjust displayed edge 2330a to a second updated edge 2330c, which is of sufficient size to encompass the relevant portion of artery 2334.

[0172] In addition to the above Figure 13A and 13B In addition to or instead of the absorption properties described, tissues and / or structures can also be imaged or characterized based on their reflectance properties at the EMR wavelength spectrum. For example, Figures 13C to 13E Various graphs showing the reflectivity of different types of tissues or structures at different EMR wavelengths. Figure 13C is a graphical representation 1050 of an exemplary ureteral feature relative to a mask. Figure 13D is a graphical representation 1052 of an exemplary artery feature relative to an obscuration. Figure 13E is a graphical representation 1054 of an exemplary neural feature relative to an obstruction. Figures 13C to 13E The curves in FIG represent the reflectance of specific structures (ureters, arteries, and nerves) relative to the corresponding reflectance of fat, lung tissue, and blood at corresponding wavelengths as a function of wavelength (nm). These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may have corresponding detectable reflectance characteristics that will allow identification and visualization of the tissues and / or structures.

[0173] In various cases, selected wavelengths for spectral imaging (i.e., "selective spectral" imaging) can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site. By utilizing selective spectral imaging, the amount of time required to obtain spectral images can be minimized, allowing information to be obtained in real time or near real time and utilized during surgery. In various cases, the wavelength can be selected by the clinician or by the control circuit based on the clinician's input. In some cases, the wavelength can be selected based on, for example, machine learning and / or big data that the control circuit can access via the cloud.

[0174] The aforementioned application of spectral imaging to tissue can be used intraoperatively to measure the distance between a waveform emitter and a critical structure obscured by tissue. In one aspect of the present disclosure, referring now to Figure 14 and Figure 15 , showing a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125. In some cases, the time-of-flight sensor system 1104 may be incorporated into the surgical visualization system 100 ( Figure 1 ). Time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. Transmitted wave 1124 extends from transmitter 1106 to critical structure 1101, and received wave 1125 is reflected back from critical structure 1101 by receiver 1108. Surgical device 1102 is positioned through a trocar 1110 that extends into a cavity 1107 of a patient.

[0175] The waveforms 1124, 1125 are configured to be able to penetrate the obscured tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or a photoacoustic signal may be emitted from the emitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, the waveforms 1124, 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, the emitter 1106 is configured to be able to provide a binary signal on and off, such as Figure 15 As shown, for example, the binary signal may be measured by receiver 1108 .

[0176] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to be able to determine the distance d ( Figure 14 ). Figure 14The time-of-flight timing diagram 1130 of the transmitter 1106 and receiver 1108 is shown in Figure 15 The delay is a function of the distance d, and the distance d is given by:

[0177]

[0178] in:

[0179] c = speed of light;

[0180] t = length of the pulse;

[0181] q11 = charge accumulated when light is emitted; and

[0182] q2 = Charge accumulated when no light is emitted.

[0183] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to Figure 14 In various cases, additional transmitter / receivers and / or pulsed signals from transmitter 1106 can be configured to transmit non-penetrating signals. Non-tissue penetrating signals can be configured to determine the distance from the transmitter to the surface 1105 of the obscuring tissue 1103. In various cases, the depth of the critical structure 1101 can be determined by the following formula:

[0184] d A =d w -d t .

[0185] in:

[0186] d A = depth of key structure 1101;

[0187] d w = distance from transmitter 1106 to key structure 1101 ( Figure 14 d) in the above clause; and

[0188] d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102 ) to the surface 1105 of the shielding tissue 1103 .

[0189] In one aspect of the present disclosure, see now Figure 16 , shows a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c. In some cases, the time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 ( Figure 1). The time-of-flight sensor system 1204 includes a waveform transmitter 1206 and a waveform receiver 1208. The waveform transmitter 1206 is positioned on the first surgical device 1202a, and the waveform receiver 1208 is positioned on the second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their respective trocars 1210a and 1210b, respectively, which extend into a cavity 1207 of a patient. Transmitted waves 1224a, 1224b, and 1224c extend from the transmitter 1206 toward the surgical site, and received waves 1225a, 1225b, and 1225c are reflected from various structures and / or surfaces at the surgical site back to the receiver 1208.

[0190] Different emission waves 1224a, 1224b, 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c can be in the visible light, NIR, or SWIR wavelength spectrum. For example, visible light can be reflected from the surface 1205 of the tissue 1203, and NIR and / or SWIR waveforms can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, as described herein, a spectral signal (e.g., a hyperspectral, multispectral, or selective spectral) or a photoacoustic signal can be emitted from the emitter 1206. In various cases, waves 1224b, 1224c may be selected to target critical structures 1201a, 1201b within tissue 1203 based on spectral characteristics of the critical structures 1201a, 1201b, as further described herein. Photoacoustic imaging is further described in various US patent applications, which are incorporated by reference into this disclosure.

[0191] The transmitted waves 1224a, 1224b, 1224c may be reflected from the target material (ie, the surface 1205, the first critical structure 1201a, and the second critical structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c may be due to Figure 16 The distance d shown 1a d 2a d 3a d 1b d 2b d 2c and was delayed.

[0192] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 are independently positionable (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d may be calculated based on the known positions of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when the surgical devices 1202a, 1202b are robotically controlled, these positions may be known. Knowledge of the positions of the emitter 1206 and receiver 1208 and the time at which the photon stream is targeted at a certain tissue and the information of the specific response received by the receiver 1208 may allow the distance d to be determined. 1a d 2a d 3a d 1b d 2b d 2c In one aspect, the distance to the obscured critical structures 1201a, 1201b can be triangulated using the penetration wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0193] Still see Figure 16 In various cases, the receiver 1208 can be rotated so that the center of mass of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b, in the view provided to the clinician. Such an orientation can quickly convey one or more relevant distances and / or viewing angles with respect to the critical structure. For example, Figure 16 As shown, the surgical site is displayed from a perspective of key structures 1201a perpendicular to the viewing plane (i.e., with the vessels oriented in-page / out-of-page). In various cases, such an orientation may be the default setting; however, the view may be rotated or otherwise adjusted by the clinician. In some cases, the clinician may switch between different surfaces and / or target structures that define the perspective of the surgical site provided by the imaging system.

[0194] In various cases, the receiver 1208 may be mounted on a trocar or cannula (such as trocar 1210b), for example, through which the surgical device 1202b is positioned. In other cases, the receiver 1208 may be mounted on a separate robotic arm whose three-dimensional position is known. In various cases, the receiver 1208 may be mounted on a movable arm separate from the robot that controls the surgical device 1202a, or may be mounted to an operating room (OR) table that can be registered with the robot's coordinate plane during surgery. In such cases, the positions of the transmitter 1206 and the receiver 1208 may be registered with the same coordinate plane, so that distances can be triangulated based on the output from the time-of-flight sensor system 1204.

[0195] Combining a time-of-flight sensor system and near-infrared spectroscopy (NIRS), known as TOF-NIRS, which is capable of measuring time-resolved signatures of NIR light with nanosecond resolution, is described in an article entitled “TIME-OF-FLIGHT NEAR-INFRAREDSPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY INGRAPEFRUIT” (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228), which is incorporated herein by reference in its entirety and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0196] In various cases, the time-of-flight spectral waveform is configured to determine the depth of a critical structure and / or the proximity of a surgical device to a critical structure. In addition, various surgical visualization systems disclosed herein include a surface mapping logic component that is configured to create a three-dimensional rendering of the surface of visible tissue. In such cases, the clinician can be aware of the proximity (or lack thereof) of the surgical device to the critical structure even when visible tissue obstructs the critical structure. In one case, the topography of the surgical site is provided on a monitor by the surface mapping logic component. If the critical structure is close to the surface of the tissue, spectral imaging can convey the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on the known limitations of the spectral imaging system, the system is configured to convey that the critical structure is out of range even if the spectral imaging system cannot detect the critical structure at all. Therefore, the clinician can continue to move the surgical device and / or manipulate the tissue. When the critical structure moves into the range of the spectral imaging system, the system can identify the structure and therefore convey that the structure is within range. In such cases, an alert can be provided when a structure is initially identified and / or further moved within a predefined proximity zone. In such cases, proximity information (i.e., lack of proximity) can be provided to the clinician even if the spectral imaging system does not identify a critical structure using known boundaries / ranges.

[0197] Various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and alert clinicians before damaging critical structures through accidental dissection and / or transection. In various aspects, the surgical visualization system is configured to identify, for example, one or more of the following critical structures: ureters, intestines, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve and their branches), blood vessels (including the pulmonary artery and lobar arteries and pulmonary veins and lobar veins, the inferior mesenteric artery [IMA] and its branches, the superior rectal artery, the sigmoid artery and the left colic artery), the superior mesenteric artery (SMA) and its branches (including the middle colic artery, the right colic artery, the ileal artery), the hepatic artery and its branches, the portal vein and its branches, the splenic artery / vein and its branches, the external and internal (lower abdominal) ileal vessels, the short gastric arteries, the uterine arteries, the middle sacral vessels, and lymph nodes. In addition, the surgical visualization system is configured to indicate the proximity of a surgical device to a critical structure and / or alert clinicians when a surgical device is in proximity to a critical structure.

[0198] Various aspects of the present disclosure provide for intraoperative critical structure identification (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, which enable visualization of critical structures, such as those below the surface of tissue (e.g., 1.0 cm to 1.5 cm below the surface of the tissue). In other cases, the surgical visualization system may identify structures less than 1.0 cm or greater than 1.5 cm below the surface of the tissue. For example, a surgical visualization system that can identify structures that are only within 0.2 mm of the surface may be valuable if the structure is not otherwise visible due to depth. In various aspects, the surgical visualization system may enhance the clinician's view, for example, by utilizing a virtual depiction of the critical structure as a visible white light image superimposed on the surface of the visible tissue. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and may, for example, provide a proximity alert when the distal tip of the surgical instrument moves within a specific range of the critical structure (e.g., within 1.0 cm of the critical structure).

[0199] Various surgical visualization systems disclosed herein can identify when dissection is too close to a critical structure. Dissection may be "too close" to a critical structure based on temperature (i.e., too much heat near a critical structure that could risk damaging / heating / melting the critical structure) and / or tension (i.e., too much tension near a critical structure that could risk damaging / tearing / pulling the critical structure). For example, such surgical visualization systems can facilitate dissection around a vessel when skeletonizing the vessel prior to ligation. In various cases, a thermal imaging camera can be utilized to read the heat at the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the tool temperature is above a predefined threshold (such as, for example, 120°F), a warning can be provided to the clinician at a first distance (such as, for example, 10 mm), and if the tool temperature is less than or equal to the predefined threshold, a warning can be provided to the clinician at a second distance (such as, for example, 5 mm). The predefined thresholds and / or warning distances can be default settings and / or programmable by the clinician. Additionally or alternatively, the proximity alert may be associated with a thermal measurement made by the tool itself, such as a thermocouple measuring heat in the distal jaws of, for example, a monopolar or bipolar dissector or vessel sealer.

[0200] The various surgical visualization systems disclosed herein can provide sufficient sensitivity with respect to critical structures and specificity to enable clinicians to confidently perform rapid but safe dissections based on standards of care and / or device safety data. The systems can function in real time during surgical procedures intraoperatively with minimal, and in various cases, no, ionizing radiation risk to the patient or clinician. In contrast, during fluoroscopy, the patient and clinician can be exposed to ionizing radiation via, for example, an X-ray beam used to view anatomical structures in real time.

[0201] The various surgical visualization systems disclosed herein can be configured to, for example, detect and identify one or more desired types of critical structures in a forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more desired types of critical structures, for example, in the surrounding area of ​​the surgical device and / or in multiple planes / dimensions.

[0202] The various surgical visualization systems disclosed herein can be easy to operate and / or interpret. Furthermore, the various surgical visualization systems can incorporate an "override" feature that allows a clinician to override default settings and / or operations. For example, a clinician can selectively disable alerts from the surgical visualization system and / or move closer to a critical structure than the surgical visualization system suggests, such as when the risk to the critical structure is less than the risk of avoiding the area (e.g., when removing cancer surrounding a critical structure, the risk of leaving cancerous tissue behind may be greater than the risk of causing damage to the critical structure).

[0203] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, the specific implementation of a surgical visualization system may not change the way surgical procedures are performed. Furthermore, a surgical visualization system may be more economical compared to the cost of accidental transections. Data suggests that reduced accidental damage to critical structures can drive incremental compensation.

[0204] The various surgical visualization systems disclosed herein can operate in real time or near real time and far enough in advance to enable the clinician to anticipate critical structures. For example, the surgical visualization system can provide sufficient time to "slow down, assess, and avoid" in order to maximize the efficiency of the surgical procedure.

[0205] The various surgical visualization systems disclosed herein may not require contrast agents or dyes to be injected into tissue. For example, spectral imaging can be configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other cases, contrast agents can be more easily injected into the appropriate tissue layers than with other visualization systems. For example, the time between contrast agent injection and visualization of critical structures can be less than two hours.

[0206] The various surgical visualization systems disclosed herein can be associated with clinical data and / or device data. For example, the data can provide boundaries regarding how close an energy-enabled surgical device (or other potentially damaging device) should be to tissue that the surgeon does not want to damage. Any data modules that interact with the surgical visualization systems disclosed herein can be provided integrally with or separately from a robot to enable use with standalone surgical devices, such as in open or laparoscopic surgery. In various cases, the surgical visualization system can be compatible with a robotic surgical system. For example, visualization images / information can be displayed on a robotic console.

[0207] In various cases, the clinician may not know the position of the critical structure relative to the surgical tool. For example, when the critical structure is embedded in the tissue, the clinician may not be able to determine the position of the critical structure. In some cases, the clinician may want to keep the surgical device outside the range of the orientation around the critical structure and / or away from the visible tissue covering the hidden critical structure. When the orientation of the hidden critical structure is unknown, the clinician may have the risk of moving too close to the critical structure, which may cause unintentional trauma to the critical structure and / or dissection and / or excessive energy, heat and / or tension near the critical structure. Alternatively, the clinician may keep too far away from the suspected position of the critical structure and have the risk of affecting the tissue in an attempt to avoid the critical structure at a less ideal position.

[0208] The present invention provides a surgical visualization system that presents tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of a surgical device relative to a critical structure. Such tracking can occur intraoperatively, in real time, and / or near real time. In various cases, the tracking data can be provided to a clinician via a display screen (e.g., a monitor) of an imaging system.

[0209] In one aspect of the present disclosure, a surgical visualization system includes a surgical device including an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect an embedded structure and the structured light pattern on the visible surface; and a control circuit in signal communication with the camera and the imaging system, wherein the control circuit is configured to determine a distance from the surgical device to the embedded structure and provide a signal indicating the distance to the imaging system. For example, the distance can be determined by calculating the distance from the camera to a critical structure illuminated using fluoroscopy and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses of the camera (e.g., a left lens and a right lens). For example, the distance from the surgical device to the critical structure can be triangulated based on the known positions of the surgical device and the camera. Alternative devices for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue that overlays / covers the embedded critical structure. For example, the surgical visualization system can identify and enhance the view of hidden critical structures by drawing a schematic representation of the hidden critical structures on the visible structures (such as a line on the surface of the visible tissue). The surgical visualization system can also determine the distance to the enhanced line on the visible tissue.

[0210] By providing clinicians with up-to-date information about the proximity of surgical devices to hidden critical structures and / or visible structures, as provided by various surgical visualization systems disclosed herein, clinicians can make more informed decisions regarding the placement of surgical devices relative to hidden critical structures. For example, clinicians can view the distance between surgical devices and critical structures in real time / intraoperatively, and in some cases, alerts and / or warnings can be provided by the imaging system when the surgical device moves within a predefined proximity and / or zone of a critical structure. In some cases, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly" zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more of the critical structure). In such cases, clinicians can maintain momentum throughout the surgical procedure without having to monitor the suspected location of critical structures and the proximity of the surgical device thereto. As a result, certain surgical procedures can be performed faster, with fewer pauses / interruptions, and / or with improved accuracy and / or certainty. In one aspect, surgical visualization systems can be used to detect tissue variability, such as variability of tissue within an organ, to differentiate between tumors / cancer tissue / unhealthy tissue and healthy tissue. Such surgical visualization systems can maximize the removal of unhealthy tissue while minimizing the removal of healthy tissue.

[0211] Surgical Hub System

[0212] The various visualization or imaging systems described herein can be incorporated into a surgical hub system, such as in conjunction with Figure 17-19 are shown and described in further detail below.

[0213] See also Figure 17 , a computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. In one example, Figure 17 , the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and handheld intelligent surgical instruments 2112, which are configured to communicate with each other and / or with a hub 2106. In some aspects, the surgical system 2102 may include M number of hubs 2106, N number of visualization systems 2108, O number of robotic systems 2110, and P number of handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to one.

[0214] Figure 18 An example of a surgical system 2102 is shown for performing a surgical procedure on a patient lying flat on an operating table 2114 in a surgical operating room 2116. A robotic system 2110 is used as part of the surgical system 2102 during the surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (surgical robot), and a surgical robotic hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side cart 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision in the patient's body. Images of the surgical site can be obtained through a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The robotic hub 2122 can be used to process the images of the surgical site for subsequent display to the surgeon through the surgeon's console 2118.

[0215] Other types of robotic systems can be readily adapted for use with the surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0216] Various examples of cloud-based analytics performed by the cloud 2104 and suitable for use in the present disclosure are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0217] In various aspects, imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0218] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0219] One or more illumination sources can be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and can be referred to as visible light or simply light. A typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0220] The invisible spectrum (i.e., the non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0221] In various aspects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopy, angiography, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0222] In one aspect, an imaging device employs multispectral monitoring to discern topography and underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible range, such as IR and ultraviolet. Spectral imaging can allow for the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated by reference into this disclosure. Multispectral monitoring can be a useful tool for repositioning the surgical field after completing the surgical task to perform one or more of the previously described tests on the treated tissue.

[0223] It is self-evident that during any surgery, the operating room and surgical equipment need to be strictly sterilized. The strict hygiene and sterilization conditions required in a "surgical room" (i.e., an operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including the imaging device 2124 and its attachments and components. It should be understood that the sterile field can be considered to be a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered to be the area around the patient that has been prepared for a surgical procedure. The sterile field can include appropriately dressed, scrubbed team members, and all equipment and fixtures in the area.

[0224] In various aspects, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically positioned relative to the sterile field, such as Figure 18 In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 2108 are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0225] like Figure 18 As shown in FIG, a primary display 2119 is positioned within the sterile field so as to be visible to an operator at the operating table 2114. Additionally, a visualization tower 21121 is positioned outside the sterile field. Visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109, facing away from each other. A visualization system 2108, directed by hub 2106, is configured to coordinate information flow to operators inside and outside the sterile field using displays 2107, 2109, and 2119. For example, hub 2106 can cause visualization system 2108 to display a snapshot of the surgical site recorded by imaging device 2124 on non-sterile display 2107 or 2109, while simultaneously maintaining a real-time feed of the surgical site on primary display 2119. The snapshots on non-sterile display 2107 or 2109 can allow a non-sterile operator to, for example, perform diagnostic procedures related to a surgical procedure.

[0226] In one aspect, the hub 2106 is further configured to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 to the primary display 2119 within the sterile field, where it can be viewed by a sterile operator at the operating table. In one example, the input can be a modified form of a snapshot displayed on the non-sterile display 2107 or 2109, which can be routed to the primary display 2119 via the hub 2106.

[0227] See also Figure 18 , surgical instruments 2112 are used during a surgical procedure as part of the surgical system 2102. The hub 2106 is also configured to coordinate the flow of information to the displays of the surgical instruments 2112, as described in various U.S. patent applications, which are incorporated by reference into this disclosure. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 can be routed by the hub 2106 to the surgical instrument display 2115 within the sterile field, where the operator of the surgical instrument 2112 can observe the input or feedback. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0228] Figure 19 A computer-implemented interactive surgical system 2200 is shown. The computer-implemented interactive surgical system 2200 is similar in many respects to the computer-implemented interactive surgical system 2100. The surgical system 2200 includes at least one surgical hub 2236 in communication with a cloud 2204, which may include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating room devices, such as, for example, smart surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As Figure 19 As shown in the example of FIG, a surgical hub 2236 is coupled to an imaging module 2238 (which is coupled to an endoscope 2239), a generator module 2240 coupled to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communications module 2230, a processor module 2232, a storage array 2234, smart devices / instruments 2235 optionally coupled to a display 2237, and a contactless sensor module 2242. Operating room devices are coupled to cloud computing resources and data storage via the surgical hub 2236. The robotic hub 2222 can also be connected to the surgical hub 2236 and cloud computing resources. Devices / instruments 2235, visualization system 2209, etc. can be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols, as described herein. The surgical hub 2236 can be coupled to a hub display 2215 (e.g., monitor, screen) to display and overlay images received from imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display can also display data received from devices connected to the modular control tower in conjunction with images and overlays.

[0229] Situational Awareness

[0230] The various visualization systems or aspects of the visualization systems described herein may be used as part of a situational awareness system that may be provided by the surgical hub 2106, 2236 ( Figure 17-19 ) to be implemented or executed. In particular, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and motion), tissue, structures, users, and other objects located within the surgical field or operating room can provide contextual data that the situational awareness system can utilize to infer the type of surgical procedure being performed or a step thereof, the type of tissue and / or structure being manipulated by the surgeon, and the like. The situational awareness system can then utilize this contextual data to provide alerts to the user, advise the user on subsequent steps or actions, prepare the surgical device for use (e.g., activating an electrosurgical generator in anticipation of using an electrosurgical instrument in a subsequent step of the surgical procedure), intelligently control the surgical instrument (e.g., customizing surgical instrument operating parameters based on the specific health status of each patient), and the like.

[0231] While "smart" devices that include control algorithms that respond to sensed data may be an improvement over "dumb" devices that operate without regard to the sensed data, some sensed data may be incomplete or uncertain when considered in isolation, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context (e.g., knowing the type of tissue being operated on or the type of surgery being performed), the control algorithm may incorrectly or suboptimally control the modular device given specific context-free sensed data. The modular device may include any surgical device that can be controlled by the situational awareness system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal approach for controlling a control algorithm for a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and therefore respond differently to actions taken by the surgical instrument. Therefore, it may be desirable for the surgical instrument to take different actions even when sensing the same measurement value for a particular parameter. As a specific example, the optimal way to control a surgical stapling and cutting instrument in response to the instrument sensing an unexpectedly high force for closing its end effector will vary depending on whether the tissue type is prone to tearing or resistant to tearing. For tissue that is prone to tearing (such as lung tissue), the instrument's control algorithm will optimally gradually reduce the motor speed in response to the unexpectedly high force for closing, thereby avoiding tearing the tissue. For tissue that is resistant to tearing (such as gastric tissue), the instrument's control algorithm will optimally gradually increase the motor speed in response to the unexpectedly high force for closing, thereby ensuring that the end effector is properly clamped on the tissue. Without knowing whether lung tissue or gastric tissue has been clamped, the control algorithm may make suboptimal decisions.

[0232] One solution utilizes a surgical hub that includes a system configured to derive information about the surgical protocol being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about the surgical protocol from the received data, and then control the modular devices paired with the surgical hub based on the context of the inferred surgical protocol. Figure 20A diagram of a situational awareness surgical system 2400 according to at least one aspect of the present disclosure is shown. In some examples, data sources 2426 include, for example, modular apparatus 2402 (which may include sensors configured to detect parameters associated with a patient and / or the modular apparatus itself), database 2422 (e.g., an EMR database containing patient records), and patient monitoring devices 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0233] The surgical hub 2404 (which may be similar to the hub 106 in many respects) may be configured to be able to derive contextual information related to the surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data source 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity that is the subject of the procedure. Some aspects of the surgical hub 2404's ability to derive or infer information related to the surgical procedure from the received data may be referred to as "situational awareness." In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to the surgical procedure from the received data.

[0234] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data source 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular devices 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another example, the situational awareness system can include a lookup table that stores pre-characterized contextual information about the surgical procedure associated with one or more inputs (or input ranges) corresponding to the contextual information. In response to a query utilizing the one or more inputs, the lookup table can return corresponding contextual information used by the situational awareness system to control the modular devices 2402. In one example, the contextual information received by the situational awareness system of the surgical hub 2404 is associated with a specific control adjustment or set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 2402 when provided with contextual information as input.

[0235] The surgical hub 2404 incorporating a situational awareness system provides numerous benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or use of the data during the surgical procedure. Returning to the previous example, the situational awareness surgical hub 2404 can determine the type of tissue being operated on; thus, when an unexpectedly high force is detected on the end effector of a closing surgical instrument, the situational awareness surgical hub 2404 can appropriately ramp up or ramp down the motor speed of the surgical instrument appropriate for the tissue type.

[0236] As another example, the type of tissue being operated on can affect the adjustment of the compression rate and load threshold of the surgical stapling and cutting instrument for a specific tissue gap measurement value. The situational awareness surgical hub 2404 can infer whether the surgical procedure being performed is thoracic surgery or abdominal surgery, thereby allowing the surgical hub 2404 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument for the type of tissue.

[0237] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situational awareness surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the type of surgery. Since one type of surgery is typically performed in a specific body cavity, the surgical hub 2404 can then control the motor speed of the smoke evacuator appropriately for the body cavity in which the procedure is being performed. Thus, the situational awareness surgical hub 2404 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0238] As yet another example, the type of procedure being performed may affect the optimal energy level at which an ultrasonic surgical instrument or radio frequency (RF) electrosurgical instrument operates. For example, arthroscopic procedures require higher energy levels because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situational awareness surgical hub 2404 may determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 may then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The situational awareness surgical hub 2404 may determine the type of surgical procedure being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, based on the expected tissue profile of the surgical procedure. In addition, the situational awareness surgical hub 2404 may be configured to be able to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the entire surgical procedure, rather than just on a procedure-by-procedure basis. The situational awareness surgical hub 2404 can determine the step of a surgical procedure being performed or to be subsequently performed and then update the control algorithm for the generator and / or ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level at a value appropriate for the expected tissue type based on the surgical step.

[0239] As yet another example, data may be extracted from additional data sources 2426 to improve conclusions extracted by the surgical hub 2404 from one data source 2426. The situational awareness surgical hub 2404 may augment the data it receives from the modular devices 2402 with contextual information about the surgical procedure that has been constructed from other data sources 2426. For example, the situational awareness surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) based on video or image data received from a medical imaging device. However, in some cases, the video or image data may be inconclusive. Thus, in one example, the surgical hub 2404 may be further configured to combine physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 communicatively coupled to the surgical hub 2404). Figure 2 )) to determine the integrity of the suture or tissue weld. In other words, the situational awareness system of the surgical hub 2404 can consider physiological measurement data to provide additional context when analyzing visualization data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0240] Another benefit includes actively and automatically controlling the paired modular devices 2402 according to the specific steps of the surgical procedure being performed, thereby reducing the number of times medical personnel need to interact with or control the surgical system 2400 during the surgical procedure. For example, if the situational awareness surgical hub 2404 determines that a subsequent step in the procedure requires the use of an RF electrosurgical instrument, it can actively activate the generator connected to the instrument. Actively activating the energy source allows the instrument to be ready for use as soon as the previous step of the procedure is completed.

[0241] As another example, the situational awareness surgical hub 2404 may determine whether the current or subsequent steps of the surgical procedure require a different view or degree of magnification on the display based on the feature(s) that the surgeon anticipates needing to view at the surgical site. The surgical hub 2404 may then proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system 108) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0242] As yet another example, the situational awareness surgical hub 2404 can determine which step of a surgical procedure is currently being performed or will subsequently be performed and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up data screens based on the step of the surgical procedure being performed, without having to wait for the surgeon to request that specific information.

[0243] Another benefit includes checking for errors during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 2404 can determine whether the operating room is correctly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve the corresponding list, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout with the standard layout determined by the surgical hub 2404 for the type of surgical procedure being performed. In one example, the surgical hub 2404 can be configured to compare a list of items for surgery (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 2404 with a recommended or expected list of items and / or devices for a given surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing if there is any discontinuity between the lists. In one example, the surgical hub 2404 can be configured to determine the relative distance or position of the modular device 2402 and the patient monitoring device 2424, for example, via a proximity sensor. The surgical hub 2404 can compare the relative position of the devices to a recommended or expected layout for a particular surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout if there is any discontinuity between the layouts.

[0244] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from an expected course of action during a surgical procedure. For example, the surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment determined by the surgical hub 2404 for the type of surgical procedure being performed. In one example, the surgical hub 2404 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0245] In general, the situational awareness system for the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) to the specific context of each surgical procedure (such as adjusting for different tissue types) and validating movements during the surgical procedure. The situational awareness system also improves the surgeon's efficiency in performing surgical procedures by automatically suggesting next steps, providing data, and adjusting displays and other modular devices 2402 in the operating room based on the specific context of the surgery.

[0246] Now see Figure 21 , which shows a depiction of a hub such as surgical hub 106 or 206 ( Figures 1 to 11 ). Timeline 2500 is an illustrative surgical procedure and contextual information that the surgical hub 106, 206 can derive from data received from data sources at each step in the surgical procedure. Timeline 2500 depicts typical steps that nurses, surgeons, and other medical staff would take during a segmentectomy surgery, starting with setting up the operating room and ending with transferring the patient to the postoperative recovery room.

[0247] The situational awareness surgical hub 106, 206 receives data from data sources throughout the surgical procedure, including data generated each time medical personnel utilize a modular device paired with the surgical hub 106, 206. The surgical hub 106, 206 can receive this data from the paired modular devices and other data sources and continuously derive inferences about the ongoing surgery (i.e., contextual information) as new data is received, such as which step of the surgery is being performed at any given time. The situational awareness system of the surgical hub 106, 206 is capable of, for example, recording data related to the procedure for generating reports, verifying the steps being taken by medical personnel, providing data or prompts that may be related to specific procedure steps (e.g., via a display screen), adjusting modular devices based on the context (e.g., activating a monitor, adjusting the field of view (FOV) of a medical imaging device, or changing the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and taking any other such actions described above.

[0248] As a first step 2502 in this exemplary operation, hospital staff retrieves the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 106, 206 determines that the operation to be performed is a thoracic operation.

[0249] In a second step 2504, staff scans the incoming medical supplies for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies utilized in various types of surgeries and confirms that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical hub 106, 206 can also determine that the procedure is not a wedge procedure (because the incoming supplies lack certain supplies required for a thoracic wedge procedure or are otherwise not consistent with a thoracic wedge procedure).

[0250] In a third step 2506, the medical personnel scans the patient belt via a scanner communicatively connected to the surgical hub 106, 206. The surgical hub 106, 206 may then confirm the identity of the patient based on the scanned data.

[0251] In a fourth step 2508, the medical staff turns on the auxiliary equipment. The auxiliary equipment utilized may vary depending on the type of surgical procedure and the techniques to be used by the surgeon, but in this exemplary case, they include a smoke evacuator, an insufflator, and a medical imaging device. When activated, as part of their initialization process, the auxiliary equipment, which is a modular device, may automatically pair with a surgical hub 106, 206 located in a specific vicinity of the modular device. The surgical hub 106, 206 may then derive contextual information about the surgical procedure by detecting the type of modular device with which it was paired during this preoperative or initialization phase. In this specific example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, the list of medical supplies used in the procedure, and the type of modular device connected to the hub, the surgical hub 106, 206 can generally infer the specific procedure that the surgical team will perform. Once the surgical hub 106, 206 knows what specific procedure is being performed, the surgical hub 106, 206 can retrieve the steps of that procedure from memory or the cloud and then cross-reference the data it subsequently receives from connected data sources (e.g., modular devices and patient monitoring devices) to infer what steps of the surgical procedure the surgical team is performing.

[0252] In step 510, the staff attaches EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hub 106, 206. When the surgical hub 106, 206 begins receiving data from the patient monitoring devices, the surgical hub 106, 206 thus confirms that the patient is in the operating room.

[0253] In step 2512, medical personnel induce anesthesia in the patient. The surgical hub 106, 206 may infer that the patient is under anesthesia based on data from the modular device and / or the patient's monitoring device (including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof). Upon completion of step 2512, the preoperative portion of the lung segmentectomy surgery is complete, and the surgical portion begins.

[0254] In step 2514, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). For example, the surgical hub 106, 206 may infer from ventilator data that the patient's lung has collapsed. The surgical hub 106, 206 may infer that the surgical portion of the procedure has begun because it may compare the detection of the patient's lung collapsing with the expected steps of the procedure (which may have been previously accessed or retrieved) and determine that collapsing the lung is the first surgical step in this particular procedure.

[0255] In step eight 2516, a medical imaging device (e.g., an endoscope) is inserted and video from the medical imaging device is started. The surgical hub 106, 206 receives the medical imaging device data (i.e., video or image data) through its connection to the medical imaging device. After receiving the medical imaging device data, the surgical hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 106, 206 can determine that the particular procedure being performed is a segmentectomy, rather than a lobectomy (note that wedge procedures have been excluded based on the data received by the surgical hub 106, 206 at step two 2504 of the procedure). From the medical imaging device 124 ( Figure 2) can be used to determine contextual information related to the type of procedure being performed in a number of different ways, including by determining the angle of the medical imaging device's visualization orientation relative to the patient's anatomy, monitoring the number of medical imaging devices utilized (i.e., activated and paired with the surgical hub 106, 206), and monitoring the type of visualization devices utilized. For example, one technique for performing a VATS lobectomy places a camera in the lower anterior corner of the patient's thorax above the diaphragm, while a technique for performing a VATS segmentectomy places a camera in an anterior intercostal position relative to the segmental fissure. For example, using pattern recognition or machine learning techniques, a situational awareness system can be trained to recognize the positioning of the medical imaging device based on visualization of the patient's anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As another example, a technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure that is not used in a VATS pulmonary resection. By tracking any or all of this data from the medical imaging device, the surgical hub 106, 206 can therefore determine the specific type of surgical procedure being performed and / or the techniques used for a particular type of surgical procedure.

[0256] In step 2518, the surgical team begins the dissection step of the procedure. The surgical hub 106, 206 may infer that the surgeon is dissecting to mobilize the patient's lung because it receives data from the RF generator or ultrasonic generator indicating that an energy instrument is being fired. The surgical hub 106, 206 may intersect the received data with the search steps of the surgical procedure to determine that the energy instrument fired at this point in the procedure (i.e., after the previously discussed surgical steps have been completed) corresponds to the dissection step. In some cases, the energy instrument may be an energy tool mounted to a robotic arm of a robotic surgical system.

[0257] In step 10 2520 , the surgical team proceeds with the ligation step of the procedure. The surgical hub 106 , 206 can infer that the surgeon is ligating the artery and vein because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similar to the previous step, the surgical hub 106 , 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieval step in the process. In some cases, the surgical instrument can be a surgical tool mounted to a robotic arm of a robotic surgical system.

[0258] In step 11 2522, the segmentectomy portion of the surgery is performed. The surgical hub 106, 206 can infer that the surgeon is transecting soft tissue based on data from the surgical stapling and cutting instrument (including data from its cartridge). The cartridge data can correspond to, for example, the size or type of staples fired by the instrument. Because different types of staples are used for different types of tissue, the cartridge data can indicate the type of tissue being sutured and / or transected. In this case, the type of staple fired is for soft tissue (or other similar tissue types), which allows the surgical hub 106, 206 to infer that the segmentectomy portion of the surgery is being performed.

[0259] In step 2524, the node dissection step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 106, 206 can infer that the surgical team is dissecting a node and performing a leak test. For this particular procedure, the RF or ultrasonic instrument utilized after transecting the soft tissue corresponds to the node dissection step, which allows the surgical hub 106, 206 to make such an inference. It should be noted that surgeons regularly switch back and forth between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific steps in the procedure, as different instruments are better suited for specific tasks. Therefore, the specific sequence in which the stapling / cutting instruments and surgical energy instruments are used can indicate the steps of the procedure being performed by the surgeon. Furthermore, in some cases, robotic tools may be used for one or more steps in the surgical procedure, and / or handheld surgical instruments may be used for one or more steps in the surgical procedure. One or more surgeons may, for example, alternate between robotic tools and handheld surgical instruments and / or may use the devices simultaneously. Upon completion of step 2524, the incision is closed and the postoperative portion of the procedure begins.

[0260] Thirteenth step 2526, reverse anesthesia to the patient. For example, the surgical hub 106, 206 may infer that the patient is waking up from anesthesia based on, for example, ventilator data (ie, the patient's breathing rate begins to increase).

[0261] Finally, in step 14 2528, the medical staff removes the various patient monitoring devices from the patient. Thus, when the hub loses EKG, BP, and other data from the patient monitoring devices, the surgical hub 2106, 2236 can infer that the patient is being transferred to a recovery room. As can be seen from the description of this exemplary surgery, the surgical hub 2106, 2236 can determine or infer when each step of a given surgical procedure occurred based on data received from various data sources communicatively coupled to the surgical hub 2106, 2236.

[0262] Situational awareness is further described in various U.S. patent applications, which are incorporated by reference into this disclosure, which is incorporated by reference herein. In some cases, the operation of a robotic surgical system (including, for example, the various robotic surgical systems disclosed herein) may be controlled by the hub 2106, 2236 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 2104 ( Figure 17 ) information to control.

[0263] Resection margin determination and adjustment

[0264] The aforementioned surgical visualization systems can be used to detect critical structures or patient tissue (e.g., a tumor) to be removed from an anatomical structure (e.g., an organ). However, many surgical procedures also require a resection margin around the patient tissue or a margin of unaffected tissue to be removed. This resection margin can be determined and / or adjusted based on a variety of features of the anatomical structure, many of which are difficult to see with the naked eye. Features can include critical structures other than the patient tissue, but relevant to the resection of the patient tissue. For example, features of the anatomical structure may include: secondary anatomical structures proximal to the patient tissue (e.g., an artery or ureter), foreign structures proximal to the patient tissue (e.g., surgical devices, surgical fasteners, or clamps), tissue mass surrounding the patient tissue (e.g., tissue damaged by emphysema), and / or the physical contours of the anatomical structure (e.g., an organ wall). There is an increasing demand for surgical visualization systems that are capable of detecting such critical structures, synthesizing data associated with the patient tissue, and communicating the synthesized data to the operating clinician in the form of relevant information and / or instructions. For example, it is desirable for a surgical visualization system to detect the position of patient tissue associated with the anatomical structure, determine a resection margin around the patient tissue, and adjust the resection margin based on the detected anatomical features. Where the surgical visualization system is further configured to detect the position of the surgical instrument relative to the resection margin, the surgical visualization system can notify the operating clinician if the surgical instrument is improperly positioned before the surgical procedure begins. Thus, in various non-limiting aspects of the present disclosure, systems and methods are provided for determining, adjusting, and executing a resection margin around patient tissue based on detection of various critical structures.

[0265] For the sake of demonstration, the foregoing principles are discussed only in the context of a surgical stapler. The present disclosure can be effectively implemented in a variety of surgical systems, including those using radiofrequency (RF) energy. Therefore, these examples herein are exemplary and are not intended to limit the scope of the present disclosure.

[0266] For example, in Figure 2In a non-limiting aspect of the present invention, a control system 133 can be implemented in the surgical visualization system 100 and determine a resection margin based on detected patient tissue associated with the anatomical structure. For example, the control circuit 132 can detect various key structures based on a signal received from the image sensor 135. The signal can be associated with electromagnetic radiation emitted by the structured light source 152 and / or the spectral light source 150 and reflected from various features of the anatomical structure. The key structure detected by the control circuit 132 can be patient tissue, and the control circuit 132 can further integrate data associated with the patient tissue into a three-dimensional digital representation or model of the anatomical structure. The control circuit 132 can use any combination of surface mapping logic 136, imaging logic 138, tissue identification logic 140, distance determination logic 141 and / or modules stored in the memory 134 to determine the resection margin based on data associated with the patient tissue relative to the model of the anatomical structure. However, the present disclosure contemplates alternative components and / or methods of determining the resection margin (e.g., central processing unit, FPGA). Additionally and / or alternatively, the critical structure detected by the control circuit 132 can be a feature of the anatomical structure, and the control circuit 132 can further integrate data associated with the feature into the model of the anatomical structure. The control circuit 132 can further adjust the resection margin based on the data associated with the feature relative to the model of the anatomical structure. The display 146 of the control system 133 can depict real, virtual and / or virtual enhanced images and / or information. For example, the display 146 may include one or more screens or monitors that are configured to communicate information such as the model of the anatomical structure, the patient tissue, the resection margin and / or the adjusted resection margin to the clinician.

[0267] See again Figure 13B , depicting an example of a surgical visualization system configured to determine a resection margin 2330a and an adjusted resection margin 2330b in accordance with at least one aspect of the present disclosure. Figure 13B The surgical visualization system includes a spectral imaging device 2320 configured to emit electromagnetic radiation onto an anatomical structure. The electromagnetic radiation may constitute a pattern of structured light and / or include spectral light of multiple wavelengths. Figure 13BIn one non-limiting aspect, the spectral imaging device 2320 may emit both structured light and spectral light. For example, at least a portion of the electromagnetic radiation includes a structured pattern that is emitted onto the anatomical structure, and at least a portion of the electromagnetic radiation includes multiple wavelengths that are configured to penetrate shielding tissue of the anatomical structure and reflect from the critical structures 2332, 2338. The structured light and spectral light may be visible or invisible. However, in other non-limiting aspects, the surgical visualization system may include separate spectral imaging devices 2320, each of which emits structured light or spectral light. Similarly, although Figure 13B The surgical visualization system is simplified to minimize hardware, but otherwise includes separate, dedicated components configured to achieve the same effect.

[0268] See further Figure 13B , the spectral imaging device 2320 may further include an image sensor 135 ( Figure 2 ), the image sensor is configured to detect reflected electromagnetic radiation of various wavelengths. For example, the image sensor 135 may detect a structured pattern of electromagnetic radiation that has been reflected from the anatomical structure. As previously described, the structured pattern of electromagnetic radiation (e.g., stripes or lines) is emitted by the spectral imaging device 2320 and projected onto the surface of the anatomical structure. The structured pattern of electromagnetic radiation may include wavelengths that are configured to be reflected from the surface of the anatomical structure. Thus, the image sensor 135 may detect at least a portion of the structured pattern of electromagnetic radiation that has been reflected from the surface of the anatomical structure. Although the image sensor 135 may include photosensitive elements (such as pixels) and be configured to be capable of CCD and / or CMOS, the present disclosure encompasses other suitable image sensors and configurations.

[0269] Image sensor 135 may include photosensitive elements that may generate signals associated with reflected electromagnetic radiation via a photoelectric effect. For example, pixels of image sensor 135 may convert light-generated charge into a voltage, and then amplify and transmit the voltage to control circuit 132 ( Figure 2 ) for further processing. After receiving the signal from the image sensor 135, the control circuit 132 ( Figure 2 ) can process the signal associated with the reflected electromagnetic radiation to determine the deformation of the structured pattern. Then, the control circuit 132 ( Figure 2 ) can assess the degree of deformation relative to the initial structured pattern of electromagnetic radiation and generate a surface map of the anatomical structure based on the assessment. Although other non-limiting aspects of the present disclosure utilize alternative methods of characterizing anatomical structures, Figure 13B The surgical visualization system can determine the distance logic component 141 ( Figure 2) further determines the dimensions of the anatomical structure and via the surface mapping logic component 136 ( Figure 2 ) further defines the contours of the anatomical structure. Thus, the surgical visualization system can generate a three-dimensional model of the anatomical structure, which can be displayed via the display 146 ( Figure 2 ) to the operating clinician. Figure 13B The surgical visualization system of uses structured light to model anatomical structures, but the present disclosure contemplates other suitable methods to map tissue, such as laser detection and ranging (LIDAR) technology.

[0270] Still see Figure 13B , the image sensor 135 may further detect spectral light that has been reflected from key structures 2332, 2338 associated with the anatomical structure. For example, the image sensor 135 may detect wavelengths of electromagnetic radiation that have been reflected from patient tissue 2332 and / or features 2338 of the anatomical structure. As previously described, the spectral light portion of the emitted electromagnetic radiation includes a plurality of wavelengths. Each of the plurality of wavelengths may be selected taking into account the expected absorption coefficients possessed by the various tissues comprising the anatomical structure. The absorption coefficients affect the extent to which each of the plurality of wavelengths is reflected, refracted, and / or absorbed by various portions of the anatomical structure. This interaction between wavelength and tissue constitutes the Figure 1 and Figure 11 The "molecular response" discussed herein. Thus, the electromagnetic radiation may include multiple different wavelengths, and each wavelength may react differently with different parts of the anatomy. Thus, the electromagnetic radiation may be specifically targeted to specific critical structures at specific locations within the anatomy.

[0271] Image sensor 135 can detect reflected spectral light in a manner similar to its structured light detection. For example, image sensor 135 can detect reflected spectral light using photosensitive elements that can generate signals via the photoelectric effect. However, for spectral light, image sensor 135 and / or control circuit 132 ( Figure 2 ) may further compile the signals generated by each reflected wavelength of the plurality of wavelengths into an image associated only with the reflected wavelength. The image sensor 135 and / or the control circuit 132 may compile these images into a three-dimensional spectral cube for further processing and analysis, each of these images being associated with a signal generated by a particular reflected wavelength. For example, the spectral cube may include data associated with a first spatial dimension and a second spatial dimension of the anatomical structure, and a third spectral dimension associated with a wavelength range. The image sensor 135 and / or the control circuit 132 may generate the spectral cube using a variety of spectral imaging techniques, including spatial scanning, spectral scanning, snapshot imaging, and / or spatial spectral scanning, etc. Although Figure 13BThe surgical visualization system utilizes an image sensor 135 to identify key structures within the anatomy, but the present disclosure contemplates other means of identification, such as ultrasound and / or photoacoustic imaging.

[0272] Control circuit 132 ( Figure 2 ) can detect the position of key structures relative to the model of the anatomical structure based on a spectral cube. For example, the control circuit 132 can use the spectral cube generated by the image sensor 135 to identify patient tissue 2332 (e.g., a tumor) within an anatomical structure (e.g., an organ). The control circuit 132 can integrate data from the spectral cube associated with the patient tissue 2332 into the model of the anatomical structure. The control circuit 132 can generate relationship data, such as the position of the patient tissue 2332 relative to the anatomical structure. Therefore, the control circuit 132 can determine a resection margin 2330a of unaffected tissue surrounding the patient tissue 2332 based on this relationship data. When determining the resection margin 2330a, the control circuit 132 can account for the geometric contours of the anatomical structure based on the model. For example, if the control circuit 132 determines based on the model that the resection margin 2330a will otherwise intersect with the boundary (e.g., a wall) of the anatomical structure, the control circuit 132 can adjust the resection margin 2330a to the boundary of the anatomical structure. Similarly, when determining the resection margin 2330a, the control circuit 132 can account for other geometric features of the anatomical structure. For example, if the control circuitry 132 determines, based on the model, that the resection edge 2330a would otherwise traverse a geometric feature of the anatomical structure (eg, a fissure), the control circuitry 132 may adjust the resection edge 2330a to bypass or encompass the geometric feature.

[0273] Figure 13B The surgical visualization system of FIG. 130 may determine a resection margin 2330a based on relationship data, such as the location of the patient tissue 2332 within the anatomical structure, and the data stored in the memory 134 ( Figure 2 ). For example, the instructions may include predetermined dimensions measured from the identified boundaries of the patient tissue 2332. In other non-limiting aspects, the instructions may include predetermined or safe margins 5030 ( Figure 22 ), the control circuit 132 may apply the determined resection margin 2330b to enhance separation and removal of the patient tissue 2332. In other non-limiting aspects, the instructions may be associated with various parameters of the surgical procedure. For example, if the operating clinician inputs a parameter indicating that the anatomical structure is a lung, the control circuit 132 ( Figure 2) can automatically apply instructions for adjusting the determined resection margin 2330a to maintain the residual volume of the anatomical structure (e.g., lung volume) after removing the tumor. Similar instructions associated with various anatomical structures and / or patient tissues 2332 can be stored in the memory 134. For example, the control circuit 132 can adjust the resection margin 2330a based on instructions stored in the memory 134 and related to the size, geometry, and type of the patient tissue 2332 detected by the spectral imaging device 2320. When determining the resection margin 2330a, the control circuit 132 can automatically apply the instructions, or a list of instructions stored in the memory 134 can be presented to the surgical clinician for selection via the user interface of the surgical visualization system. The surgical clinician can also manage and / or modify the instructions stored in the memory 134 via the user interface of the surgical visualization system. For example, the user interface can include a keyboard, a mouse, a touch screen, a wireless device, voice commands, and / or any other suitable method for providing instructions to the surgical visualization system.

[0274] In some aspects, the surgical visualization system can perform statistical analysis to characterize the anatomical structure. For example, the surgical visualization system can perform a Procrustes analysis to characterize the shape of the anatomical structure by comparing three-dimensional scans of bone features of the anatomical structure to establish relative dimensions and distances. The bone features may include characteristics of the anatomical structure, i.e., no material deformation will occur when the anatomical structure is translated, rotated, and / or scaled. Physical markers (such as rigid fiducial markers) can be used to establish geometric data points throughout the anatomical structure and help convert the three-dimensional coordinate system into the coordinate system of the surgical visualization system. Subsequent affine transformations can be performed to characterize the lines, points, and planes of the anatomical structure, thereby accounting for the deformable soft tissue features of the anatomical features. The soft tissue features can be integrated into the Procrustes model to complete the three-dimensional model of the anatomical structure.

[0275] Figure 13B The surgical visualization system may further depict a control system 133 ( Figure 2 ) of the display 146( Figure 2 ), patient tissue 2332, and resection margin 2330a on the anatomical structure. Thus, the surgical visualization system can generate a model of the anatomical structure, detect the patient tissue 2332 within the anatomical structure, determine the resection margin 2330a around the patient tissue 2332, and communicate this information to the surgical clinician to enhance separation and removal of the patient tissue 2332 from the anatomical structure.

[0276] See further Figure 13B , the surgical visualization system may detect additional critical structures within the anatomy and determine an adjusted resection margin 2330b based on the detected additional critical structures. The control circuit 132 ( Figure 2) can detect additional critical structures, such as features of anatomical structures, based on the reflected spectral light detected by the image sensor 135. The critical structures 2338, 2334 shown may include critical structures other than the patient tissue 2332, but associated with the resection of the patient tissue. For example, one such feature 2338 of the anatomical structure may include damaged tissue surrounding the patient tissue 2332. If the tissue within the anatomical structure is damaged, the damaged tissue may have a higher absorption coefficient and, therefore, a reduced refractive index. As a result, the image sensor 135 may detect less electromagnetic radiation reflected from the damaged tissue, and the control circuit 132( Figure 2 ) can identify the damaged tissue as a feature 2338 of the anatomical structure. The control circuit 132 ( Figure 2 ) The position of the detected feature 2338 can be integrated into a model of the anatomical structure and an adjusted resection margin 2330b can be determined based on the position of the feature 2338 relative to the patient tissue 2332. Figure 13B While aspects of the present invention utilize spectral light to detect additional critical structures, other non-limiting aspects may utilize structured light to achieve the same effect. Structured light may be particularly useful when detecting critical structures located at or near the surface of an anatomical structure.

[0277] although Figure 13B The resection margin 2330a is initially determined by the control circuit 132 of the surgical visualization system ( Figure 2 ), but in other aspects of the present disclosure, the resection margin 2330a may be determined by the operating clinician and displayed via the display 146 ( Figure 2 ) is set as input.

[0278] Still see Figure 13B , the initially determined resection margin 2330a surrounds patient tissue 2332 (e.g., a tumor), but passes through a feature 2338 of the anatomical structure (e.g., damaged tissue). The damaged tissue may include any tissue that is in a condition that may hinder the ability of the anatomical structure to recover from the surgical procedure. For example, the damaged tissue may include diseased tissue, infected tissue, tissue containing adhesions, tissue affected by emphysema, and / or tissue suffering from reduced blood flow, among other conditions. The damaged tissue may be stiff and / or have poor integrity, thereby making the patient susceptible to postoperative complications. Therefore, the control circuit 132 ( Figure 2) can determine an adjusted resection margin 2330b to account for features 2338, such as damaged tissue. For example, the adjusted resection margin 2330b can be wider than the initial resection margin 2330a, encompassing not only the patient tissue 2332 but also the features 2338. Thus, both the patient tissue 2332 (e.g., a tumor) and the features 2338 (e.g., damaged tissue) can be removed from the anatomical structure (e.g., an organ), thereby promoting more effective recovery for the patient.

[0279] In other aspects, the surgical visualization system may include a laser emitter configured to emit a beam of photons at blood cells of tissue of an anatomical structure. The image sensor 135 may be further configured to detect a frequency shift of photons reflected from the blood cells of the tissue, and the control circuit 132 ( Figure 2 ) can be further configured to analyze the frequency shift and determine the quality of vascular flow through tissue surrounding the patient tissue. Thus, the surgical visualization system can further assess the integrity of the tissue and factor the integrity of the tissue into determining the adjusted resection margin 2330b. For example, if the vascular flow through the tissue is low, the tissue may have low integrity, be susceptible to tearing, and potentially lead to postoperative complications. Thus, the surgical visualization system can determine an adjusted resection margin 2330b that encompasses the tissue surrounding the patient tissue having low vascular flow.

[0280] In some aspects, the adjusted resection margin 2330b is determined based on optimization of the residual mass of the anatomical structure while completely removing the patient tissue 2332 as well as the features 2338 of the anatomical structure. For example, if the anatomical structure includes a lung, the surgical clinician may wish to remove most of the tissue damaged by emphysema as well as the targeted tumor to reduce postoperative air leakage from the lung. However, if the surgical clinician removes too much tissue, they may not be able to maintain adequate lung volume, thereby increasing the risk of other postoperative complications. Therefore, Figure 13B The surgical visualization system of the invention can help determine the optimal resection margin by characterizing the anatomical structure, the patient tissue 2332 and any features 2338 of the anatomical structure, and applying the data stored in the control system 133 ( Figure 2 ) of the memory 134 ( Figure 2 ) in any of the instructions.

[0281] In some aspects, the surgical visualization system may depict the control system 133 ( Figure 2 ) of the display 146( Figure 2), the model of the anatomical structure, the patient tissue 2332, the resection margin 2330a, and / or the adjusted resection margin 2330b on the surgical visualization system. Thus, taking into account the other features 2338 of the anatomical structure, the surgical visualization system can determine an adjusted resection margin 2330b around the patient tissue 2332 and communicate this information to the surgical clinician to further enhance the separation and removal of the patient tissue 2332 from the anatomical structure. If the resection margin 2330a is initially determined by the surgical clinician and displayed via the display 146 ( Figure 2 ) is set as input, the surgical visualization system can still determine and depict the adjusted resection margin 2330b and allow the surgical clinician to select the adjusted resection margin via the user interface or maintain the initially determined resection margin 2330a.

[0282] therefore, Figure 13B The surgical visualization system of can use electromagnetic radiation in the form of spectral light and / or structured light to scan the anatomical structure, detect additional critical structures throughout the anatomical structure, and adjust the resection margin 2330a around the patient tissue 2332 in consideration of each additional critical structure detected. For example, the control circuit 132 ( Figure 2 ) can further detect a second feature 2334 of the anatomical structure that is associated with the resection of the patient tissue 2332. Although the second feature can include another sample of damaged tissue, the second feature can also include a feature of the anatomical structure or a second feature 2334 of the anatomical structure that is proximate to the patient tissue 2332 and / or the anatomical structure. For example, the second feature 2334 can include various key structures, such as organs, veins, nerves, tissues, and / or blood vessels. Figure 13B In a non-limiting aspect of the invention, the second feature 2334 may be an artery of the patient's tissue 2332 proximate to the anatomical structure. Once the second feature is detected, the control circuit 132 ( Figure 2 ) can integrate the position of the second feature 2334 into the model of the anatomical structure and determine a second adjusted resection margin 2330c based on the position of the second feature 2334 relative to the patient tissue 2332. Thus, the surgical visualization system can determine an adjusted resection margin 2330c that will ensure complete separation and removal of the patient tissue 2332 (e.g., a tumor).

[0283] Now see Figure 22 , illustrating a display 5020 of a surgical visualization system according to at least one aspect of the present disclosure. Figure 22 The display 5020 may depict the control system 133 ( Figure 2) and a model of the anatomical structure 5024. The anatomical structure 5024 includes unaffected tissue 5026 that is not diseased and not occupied by critical structures. The model of the anatomical structure 5024 may depict detected and / or determined features, such as patient tissue 5028, a predetermined margin 5030, a resection margin 5032, a first feature 5034 of the anatomical structure 5024, and an adjusted resection margin 5036. The control system 133 of the surgical visualization system has assigned a specific color to each of these detected features of the anatomical structure 5024, and the display 5020 may depict each of these detected features in its specifically assigned color, such as via Figure 22 The information index 5022 may depict the correlation of each specific color with information related to the detection feature specified by each specific color. For example, Figure 22 The information index 5022 associates each specific color with a textual description of a corresponding feature of the anatomical structure 5024. In other aspects, the information index 5022 associates each specific color with additional information related to the corresponding feature.

[0284] like Figure 22 As depicted, the surgical visualization system can detect patient tissue 5028 within an anatomical structure 5024. The information index 5022 of the display 5020 can indicate that the detected patient tissue 5028 is a tumor. Figure 2 ) in the memory may instruct the control circuit 132 ( Figure 2 ) applies a predetermined margin 5030 around the patient tissue 5028 based on the detected tumor mass (including its size, geometry, and / or type). Thus, the control system 133 may designate the resection margin 5030 as a particular color, and the information index 5022 may associate the particular color with additional information associated with the resection margin 5030. Taking into account the detected patient tissue 5028 and the predetermined margin 5030, the control circuitry of the surgical visualization system may determine a resection margin 5032 around the patient tissue 5028. Figure 22 In the display 5020 of FIG. 5 , a resection edge 5032 is depicted in a straight line segment around the anatomical structure 5024, corresponding to the capabilities of the intended surgical instrument. For example, the surgical instrument may be a surgical stapler configured to staple tissue prior to cutting via a linear stroke. However, if other surgical instruments are implemented, the display 5020 may alternately depict the resection edge 5032.

[0285] Figure 22 The display 5020 also depicts features 5034 of the anatomical structure 5024 detected by the surgical visualization system. Figure 22Information index 5022 of display 5020 indicates that detected feature 5034 of anatomical structure 5024 is tissue 5026 that has been damaged by emphysema. Figure 22 The initially determined resection margin 5032 passes through feature 5034 of the anatomical structure 5024 , and therefore, the control circuitry of the surgical visualization system may determine an adjusted resection margin 5036 to encompass feature 5036 , patient tissue 5028 , and the predetermined margin 5030 . Figure 22 The display 5020 depicts the adjusted resection margin 5036 via a dashed line. In some aspects, the display 5020 may allow the surgical clinician to select the initially determined resection margin 5032 or the adjusted resection margin 5036. In other aspects, the display 5020 will constrain the surgical clinician to the adjusted resection margin 5036 based on instructions stored in the memory of the control system.

[0286] Now see Figure 23A and 23B , depicting various models of anatomical structures generated by a surgical visualization system according to at least one aspect of the present disclosure. Figure 23A The anatomical structure 5036 includes patient tissue 5038 and a determined resection margin 5040 surrounding the patient tissue 5038. The anatomical structure 5036 also includes a feature 5042 of the anatomical structure 5036 and an adjusted resection margin 5044 surrounding the feature 5042 and the patient tissue 5038. For example, Figure 23A The patient tissue 5038 can be a tumor.

[0287] exist Figure 23B , the anatomical structure 5046 includes various resection margins 5048 surrounding various features 5049 of the anatomical structure 5046. For example, Figure 23B The feature 5049 of the anatomical structure 5046 can be a bronchus of the anatomical structure. In addition, in the case where the surgical clinician selects a specific resection margin 5048, Figure 23B The anatomical structure 5046 includes a plurality of indicators 5050, each of which is associated with an expected volume of the anatomical structure 5046. For example, the surgical clinician can select a resection margin 5048 based on a desired postoperative lung volume. In other aspects, the control system of the surgical visualization system can automatically determine the resection margin 5048 based on instructions stored in memory that require that the resulting volume of the anatomical structure 5046 not fall below a predetermined threshold.

[0288] Now see Figure 24A , depicts another model 5052 of an anatomical structure generated by a surgical visualization system according to at least one aspect of the present disclosure. Figure 24AA simplified model 5052 of the anatomy is depicted, including patient tissue 5054 and features 5056 of the anatomy 5056 detected by the surgical visualization system. Figure 24A In model 5052, patient tissue 5054 is a tumor and detected features 5056 include tissue damaged by emphysema.

[0289] Now see Figure 24B , depicting at least one aspect of the present disclosure Figure 24A Display 5058 of model 5052. Figure 24B The display 5058 can include a resection margin overlay with an information index 5060. The information index 5060 can include information related to features depicted in the model 5052 of the anatomical structure, such as a textual description of the depicted features and / or a recommended suture type and size for the determined margin. However, in other aspects, the information index 5060 can be configured to display any information related to features depicted in the model 5052. Figure 24B The display 5058 also depicts a predetermined margin 5062 surrounding the patient tissue 5054 and an initially determined resection margin 5064 surrounding the patient tissue 5054. The initially determined resection margin 5064 may be determined by the surgical visualization system or set by the operating clinician as input. Figure 24B In aspects of the present invention, the resection margin overlay may depict an adjusted resection margin 5066 based on detected features 5056 of the anatomical structure 5052 or emphysema. The adjusted resection margin 5066 may be determined based on instructions stored in a memory of the surgical visualization system to achieve a desired physiological effect. For example, based on the characterization of the anatomical structure and / or critical structures detected therein, the surgical visualization system may determine an optimized adjusted resection margin 5066 to remove patient tissue 5054, maintain a residual volume of the anatomical structure, and / or maintain a desired quality of the anatomical structure, such as minimal air leakage. Thus, the surgical clinician may select the initially determined resection margin 5064 or the adjusted resection margin 5066 based on the information depicted by the information index 5060 of the resection margin overlay of the display 5058.

[0290] Now see Figure 25 , depicts a three-dimensional model 5068 of an anatomical structure 5069 generated by a surgical visualization system 5067 in accordance with at least one aspect of the present disclosure. The surgical visualization system 5067 includes an imaging device 5070 having a distance sensor system 5071 having a transmitter 5072 configured to transmit electromagnetic radiation 5074 onto the anatomical structure 5069 and a receiver 5076 configured to detect reflected electromagnetic radiation 5074. Figure 25The imaging device 5070 can utilize the aforementioned spectral light, structured light, and laser Doppler techniques to identify key structures (such as tumors 5078) and generate a fully integrated model 5068 and detailed representation of the anatomical structure 5069. For example, Figure 25 The three-dimensional model 5068 may depict the anatomical structure 5069 as the upper lobe of the right lung, and may specifically depict various features of the anatomical structure 5069, such as arteries 5080, veins 5082, bronchi 5084, upper lobe bronchi 5086, right pulmonary artery 5090, and / or mainstem bronchus 5092. Figure 25 The anatomical structure 5069 is a lung, but the surgical visualization system 5067 can model a variety of anatomical structures depending on the specific implementation desired. Thus, the surgical visualization system 5067 can use spectral light, structured light, and / or laser Doppler to characterize any anatomical structure and display the detected features in detail via a three-dimensional model.

[0291] Figure 25 The surgical visualization system 5067 can provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and can provide proximity alerts when the distal tip of the surgical instrument moves within a specific range of the critical structure 5078. For example, the distance sensor system 5071 of the imaging device 5070 can be positioned on the distal tip of the surgical instrument and according to the previously described Figure 5 Thus, the transmitter 5072 can transmit electromagnetic radiation 5074 onto the surface of the anatomical structure 5069, and the receiver 5076 can detect the electromagnetic radiation 5074 that has been reflected from the surface of the anatomical structure 5069. The surgical visualization system 5067 can determine the position of the transmitter 5072 relative to the surface of the anatomical structure 5069 based on the time of flight of the electromagnetic radiation 5074, or the time between the time when the electromagnetic radiation is emitted from the transmitter 5072 and the time when the electromagnetic radiation is detected by the receiver 5076. Although Figure 25 The surgical visualization system 5067 uses a distance sensor system 5071 and time-of-flight technology to determine the position of the surgical instrument relative to the anatomical structure 5069, but other suitable components and / or technologies may be used to achieve the same effect and include the position of the surgical instrument in the three-dimensional model 5068 of the anatomical structure 5069.

[0292] Now see Figure 26 , depicting at least one aspect of the present disclosure Figure 25 Display 5093 of three-dimensional model 5068. Figure 26The display 5093 may include a resection margin overlay configured to depict a user-selected transection path 5096 and a system-suggested transection path 5104. For example, the resection margin overlay may further depict detected features (such as arteries 5080, veins 5082, and bronchi 5084), detected patient tissue (such as a tumor 5094), and / or other features based on data stored in the memory 134 ( Figure 2 ) in the instructions in the display 5093. After viewing the display 5093, the surgical clinician can determine a user-selected transection path 5096 to remove the tumor 5094 and the predetermined margin 5095. For example, the surgical clinician can determine that the user-selected transection path 5096 can optimize the residual volume of the anatomical structure 5069, such as the lung volume. Therefore, the surgical clinician can provide the user-selected transection path 5096 to the surgical visualization system 5067 via the user interface.

[0293] Figure 25 The surgical visualization system 5067 may receive a user-selected transection path 5096 via a user interface and evaluate the user-selected transection path 5096 relative to the position of any detected features of the anatomical structure 5069. For example, Figure 26 As depicted, the surgical visualization system can identify that the user-selected transection path 5096 interferes with the artery 5080, vein 5082, and bronchus 5084 of the anatomical structure 5069. Accordingly, the display 5093 can depict the anticipated interference and notify the surgical clinician. The notification can be visual, auditory, tactile, and / or any combination thereof. The display 5093 can additionally highlight features or portions of the anatomical structure 5069 that are affected by the user-selected transection path 5096 and / or render the portion of the anatomical structure 5069 that is nonviable as a result of the user-selected transection path 5096. For example, Figure 26 The display 5093 may highlight a transected portion 5098 of the artery 5080 to represent the blood supply 5100 that will be affected by the user-selected transection path 5096. The display 5093 may also highlight a portion 5102 of the anatomical structure 5069 that may be rendered nonviable by the user-selected transection path 5096 due to lack of blood or air.

[0294] Additionally and / or alternatively, Figure 26The surgical visualization system 5067 can depict on the display 5093 a system-suggested transection path 5104 that optimizes the residual volume of the anatomical structure 5069, removes patient tissue 5094, and the predetermined margin 5095, while minimizing adverse effects on detected features of the anatomical structure 5069. For example, while the system-suggested transection path 5104 can maintain a low residual volume of the anatomical structure 5069, the system-suggested transection path does not interfere with the artery 5080, vein 5082, and bronchus 5084, and still removes the tumor 5094 and the predetermined margin 5095 from the upper lobe of the lung. In some aspects, the surgical visualization system 5067 can allow the surgical clinician to select either the user-selected transection path 5096 or the system-suggested transection path 5104. In other aspects, the surgical visualization system 5067 can allow the surgical clinician to reject the system-suggested transection path 5104 and input a second user-selected transection path based on the information depicted on the display 5093.

[0295] Now see Figure 27 , depicting a display 5106 of a three-dimensional model 5108 of an anatomical structure 5110 generated by a surgical visualization system 5107 in accordance with at least one aspect of the present disclosure. The surgical visualization system 5107 may include a surgical instrument 5109 having a distance sensor system, a structured light system, a spectral light system, or any combination thereof. After viewing the display 5106, the surgical clinician may determine a user-selected transection path 5112 for removing patient tissue from the anatomical structure 5110. Figure 27 The surgical visualization system 5107 may receive a user-selected transection path 5112 via a user interface and evaluate the user-selected transection path 5112 relative to the position of any detected features of the anatomical structure 5110. For example, Figure 27 The surgical visualization system 5107 has identified that the user-selected transection path 5112 will interfere with an under-expanded portion 5114 of the anatomical structure 5110. The under-expanded portion 5114 of the anatomical structure 5110 may adversely affect the resection of patient tissue and may lead to postoperative complications, including a less-than-optimal residual volume of the anatomical structure 5110. Accordingly, the display 5106 may depict the anticipated problem and provide a notification to the operating clinician. This notification may be visual, audible, tactile, and / or any combination thereof.

[0296] Additionally and / or alternatively, Figure 27The surgical visualization system 5107 may depict a system-suggested transection path 5116 on the display 5106 that will optimize the residual volume of the anatomical structure 5110, remove patient tissue and predetermined margins, and minimize adverse effects caused by detected features of the anatomical structure 5110. For example, transection of under-expanded tissue 5114 may complicate the surgical procedure and introduce unnecessary risk. Figure 27 The system-suggested transection path 5116 guides the surgical clinician to fully expanded tissue of the anatomical structure 5110, thereby minimizing risk. In some aspects, the surgical visualization system 5107 may allow the surgical clinician to select either the user-selected transection path 5112 or the system-suggested transection path 5116. In other aspects, the surgical visualization system 5107 may allow the surgical clinician to reject the system-suggested transection path 5116 and enter a second user-selected transection path based on the information depicted on the display 5106.

[0297] In any of the foregoing aspects, the surgical instrument can be configured with a distance sensor system or other device to enable the surgical visualization system to detect the position of the surgical instrument relative to the anatomical structure. If the detected position of the surgical instrument does not conform to the selected transection path, the surgical visualization system discussed herein can also issue a notification to inform the operating clinician. The surgical visualization system can issue a visual, auditory, and / or tactile notification to the operating clinician indicating that the surgical instrument should be repositioned before commencing the surgical procedure. In some aspects, the surgical visualization system can prevent the operating clinician from performing the surgical procedure until the surgical instrument is correctly positioned according to the selected transection path depicted on the display.

[0298] Exemplary clinical applications

[0299] The various surgical visualization systems disclosed herein may be used in one or more of the following clinical applications.The following clinical applications are non-exhaustive and merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0300] The surgical visualization system disclosed herein can be used in a variety of different types of surgeries, for example, in different medical specialties such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, bariatrics / gastroenterology, and hepato-pancreatico-biliary surgery (HPB). For example, in urological surgery (such as a prostatectomy), ureters can be detected in fat, or connective tissue and / or nerves can be detected in fat. For example, in gynecological oncology surgery (such as a hysterectomy), and in colorectal surgery (such as a low anterior resection (LAR) surgery), ureters can be detected in fat and / or connective tissue. For example, in thoracic surgery (such as a lobectomy), blood vessels can be detected in the lungs or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In bariatric surgery, blood vessels can be detected in fat. For example, in HPB surgery (such as hepatectomy or pancreatectomy), blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0301] In one example, a clinician may want to remove an endometrial fibroid. Based on a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may want to know during surgery which tissues constitute part of the intestine and which tissues constitute part of the rectum. In such a case, a surgical visualization system as disclosed herein can indicate the different types of tissue (intestine vs. rectum) and convey this information to the clinician via the imaging system. In addition, the imaging system can determine the proximity of the surgical device to the selected tissue and communicate this proximity. In such a case, the surgical visualization system can provide increased surgical efficiency without serious complications.

[0302] In another example, a clinician (e.g., a gynecologist) may stay away from certain anatomical areas to avoid getting too close to critical structures, and therefore, the clinician may not be able to remove, for example, all of the endometriosis. A surgical visualization system as disclosed herein may enable a gynecologist to reduce the risk of getting too close to critical structures, allowing the gynecologist to get close enough with a surgical device to remove all of the endometriosis, which may improve patient outcomes (democratizing surgery). Such a system may enable a surgeon to "keep moving" during a surgical procedure rather than repeatedly stopping and restarting in order to identify areas to avoid, for example, particularly during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and given the presence and / or thickness of the tissue involved, the system may be particularly useful for hysterectomy and endometriosis surgery.

[0303] In another example, a clinician may risk dissecting a vessel too close together, and thus potentially affecting the blood supply to lobes other than the target lobe. Furthermore, anatomical variations from patient to patient may result in dissecting vessels (e.g., branches) that affect different lobes depending on the patient. A surgical visualization system as disclosed herein can enable identification of the correct vessel at the desired location, enabling the clinician to perform the dissection with appropriate anatomical certainty. For example, the system can confirm that the correct vessel is in the correct location, and the clinician can then safely divide the vessel.

[0304] In another example, due to uncertainty in the anatomy of a vessel, a clinician may perform multiple dissections before reaching the optimal location. However, it is desirable to perform the dissection at the optimal location first, as more dissections can increase the risk of bleeding. A surgical visualization system, as disclosed herein, can minimize the number of dissections by indicating the correct vessel and optimal location for dissection. For example, the ureters and cardinal ligaments are densely packed and present unique challenges during dissection. In such cases, minimizing the number of dissections may be particularly desirable.

[0305] In another example, a clinician (e.g., a surgical oncologist removing cancerous tissue) may wish to know the identification of key structures, the location of the cancer, the stage of the cancer, and / or an assessment of tissue health. Such information goes beyond what the clinician can see with the "naked eye." A surgical visualization system as disclosed herein can determine such information intraoperatively and / or communicate such information to the clinician to enhance intraoperative decision-making and improve surgical outcomes. In some cases, the surgical visualization system can be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, for example, using an endoscope or an exoscope.

[0306] In another example, a clinician (e.g., a surgical oncologist) may want to turn off one or more alerts about the proximity of surgical tools to one or more critical structures to avoid being too conservative during surgery. In other cases, the clinician may want to receive certain types of alerts such as tactile feedback (e.g., vibration / beep) to indicate proximity and / or a "no-fly zone" to keep sufficiently away from one or more critical structures. For example, a surgical visualization system as disclosed herein can provide flexibility based on the clinician's experience and / or the desired aggressiveness of the surgery. In such cases, the system provides a balance between "knowing too much" and "knowing enough" to anticipate and avoid critical structures. The surgical visualization system can help plan the next steps during surgery.

[0307] Various aspects of the subject matter described herein are set forth in the following numbered examples.

[0308] Embodiment 1. A surgical visualization system comprising a structured light emitter configured to emit a structured pattern of electromagnetic radiation onto an anatomical structure, a spectral light emitter configured to emit electromagnetic radiation comprising a plurality of wavelengths. At least one of the plurality of wavelengths is selected to penetrate a portion of the anatomical structure and reflect from patient tissue. The image sensor is configured to detect the structured pattern of electromagnetic radiation reflected from the anatomical structure and the at least one wavelength reflected from the patient tissue. The control circuit is configured to receive a first signal and a second signal from the image sensor, construct a model of the anatomical structure based on the first signal, detect a position of the patient tissue relative to the model of the anatomical structure based on the second signal, and determine a margin around the patient tissue based at least in part on the position of the patient tissue.

[0309] Example 2. The surgical visualization system of Example 1, wherein the first signal corresponds to a structured pattern of the electromagnetic radiation reflected from the anatomical structure.

[0310] Example 3. The surgical visualization system of Example 1 or 2, wherein the second signal corresponds to the at least one wavelength reflected from the patient tissue.

[0311] Embodiment 4 The surgical visualization system of any one of embodiments 1 to 3, further comprising a memory configured to store instructions. The control circuit is further configured to adjust the margin around the patient tissue based on the instructions.

[0312] Example 5. The surgical visualization system of Example 4, wherein the instructions correspond to a residual volume of the anatomical structure.

[0313] Embodiment 6. The surgical visualization system of any of Embodiments 1 to 5, wherein the control circuit is further configured to receive a third signal from the image sensor, the third signal being associated with the reflected electromagnetic radiation, detect a location of a feature of the anatomical structure based on the third signal, and determine an adjusted margin around the patient tissue based at least in part on the location of the feature.

[0314] Example 7. The surgical visualization system of Example 6, wherein the feature is damaged tissue of the anatomical structure.

[0315] Example 8. A surgical visualization system according to Example 6, wherein the feature is a second anatomical structure proximate to the patient tissue.

[0316] Example 9. The surgical visualization system of any one of Examples 6 to 8, further comprising a surgical device and a distance sensor system, the distance sensor system comprising an emitter configured to emit a beam of photons and a receiver configured to receive reflected photons. The control circuitry is further configured to receive a fourth signal from the distance sensor system, detect a current position of the surgical device relative to the model of the anatomical structure based on the fourth signal, and determine a proposed transection path for the surgical device relative to the model of the anatomical structure based at least in part on the adjusted margin.

[0317] Example 10. The surgical visualization system of Example 9, further comprising a user interface configured to receive a user-selected transection path as input to the surgical visualization system.

[0318] Example 11. A surgical visualization system according to Example 10, wherein the control circuit is further configured to be capable of issuing a notification based on at least one of the following: the patient tissue, the feature, the edge, the adjusted edge, the current position of the surgical device, the recommended transection path of the surgical device, and the user-selected transection path of the surgical device.

[0319] Example 12. The surgical visualization system according to Example 10 further includes a display configured to depict at least one of: the model of the anatomical structure, the position of the patient tissue, the edge, the position of the feature, the adjusted edge, the current position of the surgical device, the recommended transection path of the surgical device, the user-selected transection path, and the notification.

[0320] Embodiment 13. A surgical visualization system comprising a light emitter configured to emit electromagnetic radiation comprising a plurality of wavelengths and a plurality of frequencies toward an anatomical structure. A portion of the electromagnetic radiation is configured to reflect from a surface of the anatomical structure, and a portion of the electromagnetic radiation is configured to penetrate the surface of the anatomical structure and reflect from at least one of patient tissue and features of the anatomical structure. An image sensor configured to detect the portion of the electromagnetic radiation reflected from the surface of the anatomical structure. The image sensor is further configured to detect the portion of the electromagnetic radiation that penetrates the surface of the anatomical structure and reflects from at least one of the patient tissue and features of the anatomical structure. A control circuit configured to receive a signal from the image sensor and determine an edge of at least one of the patient tissue and features surrounding the anatomical structure based at least in part on the signal. The signal is associated with the reflected electromagnetic radiation.

[0321] Example 14. The surgical visualization system of Example 13, wherein the control circuit is further configured to receive a second signal from the image sensor and determine an adjusted margin around at least one of the patient tissue and the feature of the anatomical structure based at least in part on the second signal, the second signal being associated with the reflected electromagnetic radiation.

[0322] Example 15. The surgical visualization system of Example 14, wherein the second signal is specifically associated with a wavelength of the reflected electromagnetic radiation.

[0323] Example 16. The surgical visualization system of Example 14, further comprising a surgical device and a distance sensor system configured to emit a beam of photons toward the anatomical structure and detect photons reflected from the surface of the anatomical structure. The control circuitry is further configured to receive a third signal from the distance sensor system, detect a current position of the surgical device relative to the anatomical structure based on the third signal, and determine a proposed transection path for the surgical device based at least in part on the adjusted margin.

[0324] Example 17. The surgical visualization system according to Example 16 further includes a display configured to depict at least one of: the edge, the adjusted edge, the current position of the surgical device, and the recommended transection path of the surgical device.

[0325] Embodiment 18. The surgical visualization system of any one of Embodiments 13 to 17, wherein the signal is specifically associated with a frequency shift of the reflected electromagnetic radiation, and the control circuit is further configured to construct a model of the anatomical structure based on the signal.

[0326] Example 19. A surgical visualization system comprising a spectral imaging system and a structured light system, wherein the spectral imaging system is configured to emit electromagnetic radiation comprising a plurality of wavelengths, and the structured light system is configured to emit a structured pattern of electromagnetic radiation onto a surface of an anatomical structure. The spectral imaging system is configured to detect a first structure within the anatomical structure based on reflection of a first wavelength of the plurality of wavelengths. The structured light system is further configured to generate a three-dimensional model of the anatomical structure. A sensor system is configured to detect the reflected electromagnetic radiation. A control system is configured to determine an edge surrounding the first structure within the anatomical structure based on the reflection of the first wavelength of the plurality of wavelengths.

[0327] Example 20. The surgical visualization system of Example 19, wherein the spectral imaging system is further configured to detect a second structure within the anatomical structure based on a reflection of a second wavelength of the plurality of wavelengths, and wherein the control system is further configured to adjust the margin surrounding the second structure within the anatomical structure based on the reflection of the second wavelength of the plurality of wavelengths.

[0328] Although multiple forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such details. Without departing from the scope of this disclosure, many modifications, variations, changes, substitutions, combinations and equivalents to these forms may be realized, and those skilled in the art will appreciate many modifications, variations, changes, substitutions, combinations and equivalents to these forms. In addition, alternatively, the structure of each element associated with the described form may be described as a device for providing the function performed by the element. In addition, where materials for certain components are disclosed, other materials may also be used. Therefore, it should be understood that the above-mentioned specific embodiments and the appended claims are intended to encompass all such modifications, combinations and variations within the scope of the forms disclosed by the present invention. The appended claims are intended to encompass all such modifications, variations, changes, substitutions, modifications and equivalents.

[0329] The above detailed description has been described using block diagrams, flow charts, and / or examples to illustrate various forms of apparatus and / or methods. As long as such block diagrams, flow charts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flow charts, and / or examples can be implemented individually and / or collectively by a variety of hardware, software, firmware, or any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented in whole or in part in an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as any combination thereof, and that designing circuit systems and / or writing software and / or hardware code will be within the skill of those skilled in the art based on the present disclosure. In addition, those skilled in the art will recognize that the mechanisms of the subject matter described herein can be distributed as one or more program products in a variety of forms, and that the illustrative forms of the subject matter described herein are applicable regardless of the specific type of signal-bearing medium used for actual distribution.

[0330] Instructions for programming logic to perform various disclosed aspects may be stored in a memory in the system, such as dynamic random access memory (DRAM), cache, flash memory, or other memory. In addition, instructions may be distributed via a network or through other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but is not limited to a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, a flash memory, or a tangible, machine-readable storage device used when transmitting information over the Internet via an electrical signal, an optical signal, an acoustic signal, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.). Thus, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0331] As used in any aspect of this document, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more separate instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), a field programmable gate array (FPGA)), state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, or the like. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application-specific integrated circuit, electronic circuitry forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program to at least partially implement the methods and / or apparatus described herein, or a microprocessor configured by a computer program to at least partially implement the methods and / or apparatus described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and / or electronic circuitry forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). Those skilled in the art will recognize that the subject matter described herein can be implemented in analog or digital form, or some combination thereof.

[0332] As used in any aspect of this document, the term "logic" may refer to an application, software, firmware, and / or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, instruction sets, and / or data hard-coded (e.g., non-volatile) in a memory device.

[0333] As used in any aspect herein, the terms "component," "system," "module," and the like may refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0334] As used in any aspect herein, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, where a "step" refers to manipulations of physical quantities and / or logical states, which may (but need not) take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. The terms "input" and "output" are commonly used to refer to such signals as bits, values, elements, symbols, characters, terms, numbers, and the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to such quantities and / or states.

[0335] The network may include a packet-switched network. The communication devices may be capable of communicating with each other using a selected packet-switched network communication protocol. An exemplary communication protocol may include an Ethernet communication protocol that may allow communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" published by the Institute of Electrical and Electronics Engineers (IEEE) in December 2008 and / or later versions of this standard. Alternatively or additionally, the communication devices may be capable of communicating with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may be capable of communicating with each other using the Frame Relay communication protocol. The Frame Relay communication protocol may conform to or be compatible with standards published by the International Telegraph and Telephone Consultative Committee (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard entitled "ATM-MPLS Network Interworking 2.0" published by the ATM Forum in August 2001 and / or later versions thereof. Of course, different and / or later developed connection-oriented network communication protocols are also contemplated herein.

[0336] Unless otherwise expressly indicated in the above disclosure, it is understood that discussions in the above disclosure using terms such as "process," "compute," "calculate," "determine," and "display" refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities within the computer system's registers and memories and transform them into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0337] One or more components may be referred to herein as being "configured to be able to," "configurable to," "operably / operably," "suitable / adaptable to," "capable of," "conformable / conformable to," etc. Those skilled in the art will recognize that, unless the context indicates otherwise, "configured to be able to" may generally encompass components in an active state and / or components in an inactive state and / or components in a standby state.

[0338] The terms "proximal" and "distal" are used herein relative to a clinician manipulating the handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion positioned away from the clinician. It should also be understood that for brevity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein in conjunction with the accompanying drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0339] Those skilled in the art will recognize that, in general, the terms used herein, and in particular in the appended claims (e.g., the bodies of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "comprising" should be interpreted as "including, but not limited to," etc.). Those skilled in the art will also understand that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, to aid understanding, the following appended claims may contain use of the introductory phrases "at least one" and "one or more" to introduce claims. However, the use of such phrases should not be construed as implying that introducing a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); this also applies to the use of definite articles used to introduce claim recitations.

[0340] In addition, even if a specific number of claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should generally be interpreted to mean at least the recited number (e.g., a bare recitation of "two recitations," without other modifiers, generally means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention similar to "at least one of A, B, and C, etc." is used, generally, such construction is intended to have the meaning that one skilled in the art would understand the convention (e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention similar to "at least one of A, B, or C, etc." is used, generally, such construction is intended to have the meaning that one skilled in the art would understand the convention (e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art will also understand that, generally, unless the context indicates otherwise, transitional words and / or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to encompass the possibility of including one, either, or both of the terms. For example, the phrase "A or B" will generally be understood to include the possibility of "A" or "B" or "A and B."

[0341] With respect to the appended claims, those skilled in the art will understand that the operations described therein may generally be performed in any order. In addition, although various operational flow charts are presented in one or more sequences, it will be understood that the various operations may be performed in an order other than the order shown, or the various operations may be performed simultaneously. Unless the context dictates otherwise, examples of such alternative orderings may include overlapping, interleaved, interrupted, reordered, incremental, preliminary, supplemental, simultaneous, reverse, or other altered orderings. Furthermore, unless the context dictates otherwise, terms such as "responsive to," "related to," or other past tense adjectives are generally not intended to exclude such variations.

[0342] It is worth noting that any reference to "one aspect," "an aspect," "an example," or "an example" means that the specific features, structures, or characteristics described in connection with the aspect are included in at least one aspect. Therefore, the phrases "in one aspect," "in an aspect," "in an example," and "in an example" appearing in various places throughout this specification do not necessarily refer to the same aspect. Furthermore, specific features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0343] Any patent application, patent, non-patent publication, or other disclosure material referred to in this specification and / or listed in any Application Data Sheet is incorporated herein by reference to the extent that the incorporated material is inconsistent herein. Therefore, and to the extent necessary, the disclosure explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material, or portion thereof, allegedly incorporated herein by reference that conflicts with existing definitions, statements, or other disclosure material listed herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing disclosure material.

[0344] In summary, the numerous benefits resulting from the use of the concepts described herein have been described. For purposes of illustration and description, one or more specific embodiments have been provided above. These specific embodiments are not intended to be exhaustive or to limit the invention to the precise forms disclosed. The invention may be modified or varied in light of the above teachings. The form or forms selected and described are intended to illustrate the principles and practical applications, thereby enabling one of ordinary skill in the art to utilize various forms and modifications as appropriate for the particular use contemplated. The claims submitted herewith are intended to define the full scope.

Claims

1. A surgical visualization system comprising: a structured light emitter configured to emit a structured pattern of electromagnetic radiation onto an anatomical structure; a spectral light emitter configured to emit electromagnetic radiation comprising a plurality of wavelengths, wherein at least one of the plurality of wavelengths is selected to penetrate a portion of the anatomical structure and reflect from patient tissue; an image sensor configured to detect the structured pattern of electromagnetic radiation reflected from the anatomical structure and the at least one wavelength reflected from the patient tissue; and A control circuit configured to: receiving a first signal and a second signal from the image sensor; constructing a model of the anatomical structure based on the first signal; detecting a position of the patient tissue relative to the model of the anatomical structure based on the second signal; and A margin around the patient tissue is determined based at least in part on the location of the patient tissue.

2. The surgical visualization system of claim 1 , wherein: The first signal corresponds to a structured pattern of the electromagnetic radiation reflected from the anatomical structure.

3. The surgical visualization system of claim 1 , wherein: The second signal corresponds to the at least one wavelength reflected from the patient tissue.

4. The surgical visualization system of claim 1 , further comprising a memory configured to store instructions, wherein the control circuitry is further configured to adjust the margin around the patient tissue based on the instructions.

5. The surgical visualization system of claim 4, wherein: The instructions correspond to a residual volume of the anatomical structure.

6. The surgical visualization system of claim 1 , wherein: The control circuit is further configured to: receiving a third signal from the image sensor, wherein the third signal is associated with reflected electromagnetic radiation; detecting a position of a feature of the anatomical structure based on the third signal; as well as An adjusted margin around the patient tissue is determined based at least in part on the location of the feature.

7. The surgical visualization system of claim 6, wherein: The feature is damaged tissue of the anatomical structure.

8. The surgical visualization system of claim 6, wherein: The feature is a second anatomical structure proximate to the patient tissue.

9. The surgical visualization system of claim 6, further comprising: surgical devices; and A distance sensor system, comprising: an emitter configured to emit a photon beam; and a receiver configured to receive the reflected photons; Wherein, the control circuit is further configured to: receiving a fourth signal from the distance sensor system; detecting a current position of the surgical device relative to the model of the anatomical structure based on the fourth signal; and A proposed transection path for the surgical device relative to the model of the anatomical structure is determined based at least in part on the adjusted margin.

10. The surgical visualization system of claim 9, further comprising a user interface configured to receive a user-selected transection path as input to the surgical visualization system.

11. The surgical visualization system of claim 10, wherein: The control circuit is further configured to issue a notification based on at least one of: the patient tissue, the feature, the edge, the adjusted edge, the current position of the surgical device, the suggested transection path of the surgical device, and the user-selected transection path of the surgical device.

12. The surgical visualization system of claim 11 , further comprising a display configured to depict at least one of: the model of the anatomical structure, the position of the patient tissue, the margin, the position of the feature, the adjusted margin, the current position of the surgical device, the suggested transection path of the surgical device, the user-selected transection path, and the notification.

13. The surgical visualization system of claim 1 , wherein: a portion of the electromagnetic radiation from the spectral light emitter being configured to penetrate a surface of the anatomical structure and reflect from features of the anatomical structure; wherein the image sensor is further configured to detect the portion of the electromagnetic radiation that penetrates the surface of the anatomical structure and reflects from the feature of the anatomical structure; and Wherein determining the margin includes determining a margin surrounding the location of the patient tissue and the feature of the anatomical structure.

14. The surgical visualization system of claim 13, wherein: The control circuit is further configured to: receiving a third signal from the image sensor, wherein the third signal is associated with reflected electromagnetic radiation; and An adjusted margin is determined around at least one of the patient tissue and the feature of the anatomical structure based at least in part on the third signal.

15. The surgical visualization system of claim 14, wherein: The third signal is specifically associated with the wavelength of the reflected electromagnetic radiation.

16. The surgical visualization system of claim 14, further comprising: surgical devices; and a distance sensor system configured to emit a beam of photons toward the anatomical structure and detect photons reflected from the surface of the anatomical structure; Wherein, the control circuit is further configured to: receiving a fourth signal from the distance sensor system; detecting a current position of the surgical device relative to the anatomical structure based on the fourth signal; and A proposed transection path for the surgical device is determined based at least in part on the adjusted edge.

17. The surgical visualization system of claim 16, further comprising a display configured to depict at least one of: the margin, the adjusted margin, the current position of the surgical device, and the proposed transection path of the surgical device.

Images