Visualization of surgical devices

Abstract

The present invention discloses a surgical visualization system. The surgical visualization system is configured to identify one or more structures and / or determine one or more distances relative to obscuring tissue and / or the identified structure. The surgical visualization system can facilitate a surgical device to avoid the identified structure. The surgical visualization system may include a first emitter configured to emit a plurality of tissue-penetrating light waves and a second emitter configured to emit structured light onto the surface of the tissue. The surgical visualization system may also include an image sensor configured to detect reflected visible light, tissue-penetrating light, and / or structured light. The surgical visualization system may convey information about the location of one or more hidden identified structures to one or more clinicians and / or provide one or more proximity indicators.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 62 / 698,625, filed on July 16, 2018, entitled “DIGITALSURGERY IMAGING / VISUALIZATION SYSTEM,” which is incorporated herein by reference in its entirety. Background Art

[0003] Surgical systems are often combined with imaging systems that allow clinicians to view the surgical site and / or one or more portions thereof, for example, on one or more displays (such as monitors). The displays can be local to the operating room and / or remote. The imaging system may include a scope with a camera that views the surgical site and transmits the view to a display that the clinician can view. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can identify and / or convey to clinicians. For example, some imaging systems may not be able to identify certain hidden structures, physical contours, and / or dimensions in three-dimensional space during surgery. Additionally, some imaging systems may not be able to transmit and / or convey certain information to clinicians during surgery. Summary of the Invention

[0004] A surgical visualization system may include: a display screen; a surgical device configured to emit a structured light pattern onto a surface; an image sensor configured to identify a structure embedded beneath the surface; and control circuitry in signal communication with the image sensor, wherein the control circuitry is configured to: receive imaging data indicating the structured light pattern on the surface; generate a three-dimensional digital representation of the surface based on the imaging data; obtain an image of the structure and the surgical device from the image sensor; overlay the image of the structure and the surgical device with the three-dimensional digital representation of the surface on the display screen; and determine a distance from the surgical device to the structure based on the image.

[0005] A surgical visualization system may include a processor and a memory communicatively coupled to the processor, wherein the memory stores instructions executable by the processor to: receive imaging data indicating a structured light pattern on a surface; generate a three-dimensional digital representation of the surface based on the imaging data; obtain an image of an embedded structure and a surgical device from an image sensor; overlay the image of the embedded structure and the surgical device with the three-dimensional digital representation of the surface on a display screen; and determine a distance from the surgical device to a portion of the surface that covers the embedded structure.

[0006] A non-transitory computer-readable medium may store computer-readable instructions that, when executed, cause a machine to: receive imaging data indicating a structured light pattern on a surface; generate a three-dimensional digital representation of the surface based on the imaging data; obtain a three-dimensional image of an embedded structure and a surgical device from an image sensor; overlay the image of the embedded structure and the surgical device with the three-dimensional digital representation of the surface on a display screen; and determine a distance from the surgical device to the embedded structure based on the three-dimensional image.

[0007] A surgical visualization system may include a display, a first robotic tool having a three-dimensional camera, wherein the three-dimensional camera includes an image sensor. The surgical visualization system may also include a second robotic tool including a spectral light emitter configured to emit spectral light of multiple wavelengths capable of penetrating a surface and reaching structures beneath the surface, wherein the image sensor is configured to detect reflected visible light and the reflected spectral light of multiple wavelengths. The surgical visualization system may also include control circuitry in signal communication with the image sensor and the display, wherein the control circuitry is configured to: obtain a three-dimensional image of a structure and the second robotic tool from the image sensor; determine a distance from the second robotic tool to the structure based on the three-dimensional image; and provide a signal indicating the distance to the display.

[0008] A surgical visualization system may include a transmitter configured to transmit a plurality of tissue-penetrating waveforms, a receiver configured to detect the plurality of tissue-penetrating waveforms, an imaging system including a display, and control circuitry in signal communication with the receiver. The control circuitry may be configured to receive data representing an image of a concealed portion of a surgical device from the receiver and provide the image of the concealed portion of the surgical device to the display.

[0009] A surgical visualization system may include a hyperspectral camera including an emitter configured to emit a plurality of tissue-penetrating waveforms and an image sensor configured to detect the plurality of tissue-penetrating waveforms. The surgical visualization system may also include control circuitry in signal communication with the hyperspectral camera, wherein the control circuitry is configured to: receive data indicating a position of a first critical structure from the plurality of tissue-penetrating waveforms detected by the image sensor; receive data indicating a position of a second critical structure from the plurality of tissue-penetrating waveforms detected by the image sensor; and determine a distance between the first critical structure and the second critical structure.

[0010] A non-transitory computer-readable medium may store computer-readable instructions that, when executed, cause a machine to: receive data representing a first image of a first hidden structure from an image sensor; provide the first image of the first hidden structure to a display; receive data representing a second image of a second hidden structure from the image sensor; provide the second image of the second hidden structure to the display; and determine a distance between the first hidden structure and the second hidden structure.

[0011] A surgical visualization system may include: a first projector configured to emit a structured light pattern onto a surface of an anatomical structure; a second projector configured to emit spectral light of multiple wavelengths that can penetrate the anatomical structure and reach a nail line; and a control circuit that communicates signals with an image sensor, wherein the control circuit is configured to: receive structured light data from the image sensor indicating a structured light pattern on the surface of the anatomical structure; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data from the image sensor indicating a spectral image of the nail line; generate a spectral image of the nail line based on the spectral light data; and determine a distance relative to the nail line.

[0012] A surgical visualization system may include a processor and a memory communicatively coupled to the processor, wherein the memory stores instructions that, when executed by the processor: receive structured light data from an image sensor indicating a structured light pattern on a surface of an anatomical structure; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data from the image sensor indicating a spectral image of a staple line; generate a spectral image of the staple line based on the spectral light data; and determine a distance relative to the staple line.

[0013] A non-transitory computer-readable medium may store computer-readable instructions that, when executed, cause a machine to: receive structured light data from an image sensor indicating a structured light pattern on a surface of an anatomical structure; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data from the image sensor indicating a spectral image of a nail line; generate a spectral image of the nail line based on the spectral light data; and determine a distance relative to the nail line. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The novel features of the various aspects are set forth with particularity in the appended claims. However, the described aspects, both as to organization and method of operation, may be best understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0016] Figure 2 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0017] Figure 2A Control circuitry configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0018] Figure 2B Combinatorial logic circuitry configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure is shown.

[0019] Figure 2C A sequential logic circuit configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0020] Figure 3 is a description according to at least one aspect of the present disclosure Figure 1 Triangulation between surgical devices, imaging devices and critical structures to determine the depth of critical structures below the tissue surface A Schematic diagram of .

[0021] Figure 4 is a schematic diagram of a surgical visualization system configured to identify a critical structure below a tissue surface according to at least one aspect of the present disclosure, wherein the surgical visualization system includes a method for determining a depth d of the critical structure below the tissue surface. A Pulsed light source.

[0022] Figure 5is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0023] Figure 6 is a schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, wherein the surgical visualization system is configured to identify critical structures embedded within tissue.

[0024] Figure 7A and Figure 7B According to at least one aspect of the present disclosure Figure 6 A 3D camera captures a view of the key structure, where Figure 7A is the view from the left lens of the 3D camera, and Figure 7B is the view from the right lens of the 3D camera.

[0025] Figure 8 According to at least one aspect of the present disclosure Figure 6 Schematic diagram of a surgical visualization system in which a camera-key structure distance d from a three-dimensional camera to a key structure can be determined w .

[0026] Figure 9 is a schematic diagram of a surgical visualization system utilizing two cameras to determine the position of an embedded critical structure according to at least one aspect of the present disclosure.

[0027] Figure 10A is a schematic diagram of a surgical visualization system utilizing a camera that is axially moved between a plurality of known positions to determine the position of an embedded critical structure in accordance with at least one aspect of the present disclosure.

[0028] Figure 10B According to at least one aspect of the present disclosure Figure 10A Schematic diagram of a surgical visualization system in which a camera is moved axially and rotationally between multiple known positions to determine the position of embedded critical structures.

[0029] Figure 11 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0030] Figure 12 is a schematic diagram of a structured light source for a surgical visualization system according to at least one aspect of the present disclosure.

[0031] Figure 13 is a schematic diagram of a hyperspectral visualization system for imaging terrestrial features or objects according to at least one aspect of the present disclosure.

[0032] Figure 14 is a graphical representation of hyperspectral signatures of various terrestrial features or objects according to at least one aspect of the present disclosure.

[0033] Figures 15A to 15C An example of a hyperspectral visualization system for imaging a fried egg according to at least one aspect of the present disclosure is shown, wherein Figure 15A It's a photo of fried eggs. Figure 15B is a graphical representation of the hyperspectral signatures of the yolk and white portions of a fried egg, and Figure 15C is a hyperspectral image of a fried egg (shown in black and white), where the enhanced image distinguishes the yolk and egg white parts based on the hyperspectral feature data.

[0034] Figures 16 to 18 Depicted are exemplary hyperspectral recognition features for distinguishing anatomical structures from obscurants in accordance with at least one aspect of the present disclosure, wherein Figure 16 is a graphic representation of the ureteral features and obscurations, Figure 17 is a graphical representation of arterial features and obscurations, and Figure 18 is a graphical representation of neural signatures and masking.

[0035] Figure 19 is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense the distance to a critical anatomical structure according to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) positioned on a common device.

[0036] Figure 20 According to at least one aspect of the present disclosure Figure 19 Schematic diagram of the transmitted wave, received wave, and the delay between the transmitted and received waves of the NIR time-of-flight measurement system.

[0037] Figure 21 An NIR time-of-flight measurement system configured to sense the distance of different structures is shown, including a transmitter (emitter) and a receiver (sensor) on separate devices, according to one aspect of the present disclosure.

[0038] Figure 22 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a three-dimensional camera and a surgical device having a transmitter, the surgical visualization system being configured to determine a distance from the surgical device to a critical structure beneath a tissue surface.

[0039] Figure 23 The invention shows the use of at least one aspect of the present disclosure Figure 22The views of the surgical site obtained by the two-dimensional left lens and the two-dimensional right lens of the three-dimensional camera are combined to produce a three-dimensional view on the display screen, wherein the display screen also indicates the surgical site from the two-dimensional left lens and the two-dimensional right lens. Figure 22 The distance of the surgical device to critical structures is enhanced with color coding on the view of critical structures.

[0040] Figure 24 According to at least one aspect of the present disclosure Figure 23 Schematic diagram of the display screen, which depicts the Figure 22 The 3D view obtained by the 3D camera and the indication from Figure 22 The distance from the surgical device to the critical structure is shown, with the critical structure enhanced with cross-hatching in the view.

[0041] Figure 25 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a three-dimensional camera and three surgical devices, the surgical visualization system configured to determine a distance from each surgical device to a critical structure beneath a tissue surface.

[0042] Figure 26 is a schematic diagram of a screen according to at least one aspect of the present disclosure, which depicts Figure 25 The 3D view obtained by the 3D camera and the proximity spectrum indicator indicates the Figure 25 The distance from the surgical device to critical structures.

[0043] Figure 27 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a camera and a surgical device having a transmitter, the surgical visualization system being configured to determine a device-surface distance from a distal end of the surgical device to a tissue surface, a device-vessel distance from the distal end of the surgical device to a vessel below the tissue surface, and a surface-vessel distance (depth of critical structures below the tissue surface).

[0044] Figure 28 According to at least one aspect of the present disclosure, a method for selecting a Figure 27 Schematic diagram of a dial for displaying device-surface distance, device-vessel distance, or surface-vessel distance of a surgical visualization system.

[0045] Figure 29 According to at least one aspect of the present disclosure Figure 27 Schematic diagram of the screen of the surgical visualization system, which displays Figure 28 The dial is in a first position in which surface-vessel distance is selected, and wherein the screen displays a first aggregated view including data related to surface-vessel distance.

[0046] Figure 30 According to at least one aspect of the present disclosure Figure 29 Schematic diagram of the screen showing Figure 28 The dial is in a second orientation in which the device-surface distance is selected, and wherein the display shows a second aggregated view including data related to the device-surface distance.

[0047] Figure 31 According to at least one aspect of the present disclosure Figure 29 Schematic diagram of the screen, which depicts Figure 28 The dial is in a third position in which a device-vessel distance is selected, and wherein the screen displays a third aggregated view including data related to the device-vessel distance.

[0048] Figure 32 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a spectral imaging camera configured to enable identification of hidden anatomical structures and surgical devices.

[0049] Figure 33 According to at least one aspect of the present disclosure Figure 32 Schematic diagram of a screen of a surgical visualization system, wherein the screen displays an enhanced view of a surgical site including a hidden anatomical structure and a hidden surgical device, and wherein the screen further depicts a proximity spectrum indicator that conveys the proximity of the hidden surgical device relative to the anatomical structure.

[0050] Figure 34 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a spectral imaging camera configured to identify a biopsy needle in a first position relative to an embedded tumor.

[0051] Figure 35 According to at least one aspect of the present disclosure Figure 34 A schematic diagram of a screen of a surgical visualization system, wherein the screen displays an enhanced view of a surgical site including a tumor and a biopsy needle in a first orientation, and wherein the screen further depicts a proximity spectrum indicator communicating proximity of the biopsy needle to a surface of the tumor and blocking tissue in the first orientation.

[0052] Figure 36 According to at least one aspect of the present disclosure Figure 34 Schematic diagram of a surgical visualization system depicting a biopsy needle in a second orientation relative to an embedded tumor.

[0053] Figure 37 According to at least one aspect of the present disclosure Figure 35Schematic diagram of a screen showing an enhanced view of a surgical site including a tumor and a biopsy needle in a second orientation, and wherein the screen further depicts a proximity spectrum indicator that conveys the proximity of the biopsy needle to the tumor and to a surface of blocking tissue in the second orientation.

[0054] Figure 38 is a schematic diagram of a biopsy protocol according to at least one aspect of the present disclosure, in which an ultrasound device is used to identify a tumor in the thyroid gland.

[0055] Figures 39 to 41 An example of an anastomosis step during a low anterior resection (LAR) procedure of the colon according to at least one aspect of the present disclosure is depicted, wherein Figure 39 Depicting the separated circular stapler and anvil, Figure 40 Depicts a circular stapler and anvil coupled together for firing, and Figure 41 Depicted are sections of the colon sutured together after firing.

[0056] Figure 42 is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, the surgical visualization system including a spectral imaging camera configured to identify hidden surgical devices and staple lines.

[0057] Figure 43 According to at least one aspect of the present disclosure Figure 42 Schematic diagram of a screen of a surgical visualization system, wherein the screen displays an enhanced view of a surgical site including hidden staple lines and a surgical device, and wherein the screen further depicts a proximity spectrum indicator that conveys the proximity of the surgical device relative to the staple lines.

[0058] Figure 44 According to at least one aspect of the present disclosure Figure 42 Schematic diagram of a surgical visualization system in which a spectral imaging camera is configured to identify a circular stapler and anvil.

[0059] Figure 45 According to at least one aspect of the present disclosure Figure 43 Schematic diagram of a screen showing an enhanced view of a surgical site including hidden staple lines, a circular stapler, and anvil, and wherein the screen further depicts a distance between the circular stapler and the anvil and a distance between the anvil and one of the staple lines.

[0060] Figure 46 is a schematic illustration of a stomach with a bougie positioned therein during sleeve gastrectomy according to at least one aspect of the present disclosure.

[0061] Figure 47Depicted are the suturing steps during sleeve gastrectomy according to at least one aspect of the present disclosure.

[0062] Figure 48 is a schematic diagram of a surgical visualization system including a spectral imaging camera according to at least one aspect of the present disclosure, wherein the spectral imaging camera views a portion of the stomach during sleeve gastrectomy and a bougie is hidden by the stomach; however, for illustrative purposes, the stomach is partially resected to expose the bougie.

[0063] Figure 49 According to at least one aspect of the present disclosure Figure 48 Schematic diagram of a screen of a surgical visualization system, wherein the screen displays an enhanced view of a surgical site including a stomach and a bougie and a surgical stapler therein, and wherein the screen further depicts a distance between the surgical stapler and the bougie.

[0064] Figure 50 is a schematic diagram of a surgical visualization system including a spectral imaging camera according to at least one aspect of the present disclosure, wherein the spectral imaging camera is configured to identify surgical devices and clamps hidden within tissue.

[0065] Figure 51 According to at least one aspect of the present disclosure Figure 50 Schematic diagram of a screen of a surgical visualization system, wherein the screen displays an enhanced view of a surgical site including a surgical device and a clamp, wherein the screen also provides warnings based on the relative orientation of the surgical device and the clamp.

[0066] Figure 52 Depicted are steps during a hernia repair procedure in which a clip and mesh are installed at a surgical site, according to at least one aspect of the present disclosure. DETAILED DESCRIPTION

[0067] The applicant of the present application also owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety:

[0068] U.S. patent application Ser. No. 16 / 128,179, entitled “SURGICAL VISUALIZATION PLATFORM”;

[0069] U.S. Patent Application 16 / 128,191, entitled “SURGICAL VISUALIZATION CONTROLS”;

[0070] U.S. patent application Ser. No. 16 / 128,180, entitled “CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE”;

[0071] U.S. patent application Ser. No. 16 / 128,198, entitled “SINGULAR EMR SOURCE WITH DUAL OUTPUT EMITTER ASSEMBLY”;

[0072] U.S. patent application Ser. No. 16 / 128,207, entitled “COMBINATION EMITTER AND CAMERA ASSEMBLY”;

[0073] U.S. Patent Application 16 / 128,176, entitled “SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES”;

[0074] U.S. patent application Ser. No. 16 / 128,187, entitled “SURGICAL VISUALIZATION OF MULTIPLE TARGETS”;

[0075] U.S. patent application Ser. No. 16 / 128,192, entitled “VISUALIZATION OF SURGICAL DEVICES”;

[0076] U.S. patent application Ser. No. 16 / 128,163, entitled “OPERATIVE COMMUNICATION OF LIGHT”;

[0077] U.S. patent application Ser. No. 16 / 128,197, entitled “ROBOTIC LIGHT PROJECTION TOOLS”;

[0078] U.S. patent application Ser. No. 16 / 128,164, entitled “SURGICAL VISUALIZATION FEEDBACK SYSTEM”;

[0079] U.S. patent application Ser. No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MONITORING”;

[0080] U.S. patent application Ser. No. 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA”;

[0081] U.S. patent application Ser. No. 16 / 128,170, entitled “ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS”;

[0082] U.S. patent application Ser. No. 16 / 128,183, entitled “SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS”;

[0083] U.S. patent application Ser. No. 16 / 128,172, entitled “Robotic system with separate photoacoustic receiver”; and

[0084] • U.S. Patent Application 16 / 128,185, entitled “FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION.”

[0085] The applicant of the present application also owns U.S. Patent No. 9,072,535, entitled “SURGICAL STAPLING INSTRUMENTS WITH ROTATABLE STAPLE DEPLOYMENT ARRANGEMENTS,” published on July 7, 2015, which is incorporated herein by reference in its entirety.

[0086] The applicant of the present application also owns U.S. Provisional Patent Application No. 62 / 611,339, filed on December 28, 2017, entitled “ROBOT ASSISTED SURGICAL PLATFORM,” which is incorporated herein by reference in its entirety.

[0087] The applicant of the present application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0088] U.S. patent application Ser. No. 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0089] U.S. patent application Ser. No. 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICALPLATFORMS”

[0090] U.S. Patent Application 15 / 940,711, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”; and

[0091] ·Submitted on March 29, 2018, titled “CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-

[0092] CHROMATIC LIGHT REFRACTIVITY”, which is incorporated herein by reference in its entirety.

[0093] Before describing in detail various aspects of the surgical visualization platform, it should be noted that the illustrative examples are not limited in application or use to the details of construction and arrangement of components shown in the drawings and the specification. The illustrative examples may be implemented or incorporated in other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise indicated, the terms and expressions used herein are selected for the convenience of the reader and are not intended to be limiting. Furthermore, it should be understood that one or more of the aspects, expressions of aspects, and / or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and / or examples described below.

[0094] The present disclosure relates to a surgical visualization platform that utilizes "digital surgery" to obtain additional information about a patient's anatomy and / or surgical procedure. The surgical visualization platform is further configured to communicate the data and / or information to one or more clinicians in a helpful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomy and / or surgical procedure.

[0095] "Digital surgery" can encompass robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and more. Although the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In some cases, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0096] In some cases, a surgical system incorporating a surgical visualization platform can implement intelligent dissection to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve and / or tumors. In other cases, the critical structure can be, for example, an alien structure in the dissection field, such as a surgical device, surgical fastener, clamp, tack, bougie, band and / or plate. The critical structure can be determined based on different patients and / or different surgeries. Exemplary critical structures are also described herein. For example, intelligent dissection technology can provide improved intraoperative guidance for dissection and / or key anatomical structure detection and avoidance technology can be used to implement intelligent decision-making.

[0097] Surgical systems incorporating surgical visualization platforms can also implement intelligent anastomosis technology that utilizes improved workflows to provide more consistent anastomosis at optimal locations. Various surgical visualization platforms and procedures described herein can also be utilized to improve cancer localization technology. For example, cancer localization technology can identify and track cancer location, orientation, and its boundaries. In some cases, cancer localization technology can compensate for movement of tools, patients, and / or patient anatomy during surgery to provide guidance to the clinician returning to a point of interest.

[0098] In certain aspects of the present disclosure, a surgical visualization platform can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly when dissecting and / or placing suturing devices within tissue. Certain tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0099] These and other related subject matter are described herein and / or in the aforementioned concurrently filed U.S. patent applications, which are incorporated herein by reference in their entireties.

[0100] During surgery, the information available to the clinician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures (such as structures embedded or buried within an organ) may be at least partially concealed or hidden from view. Additionally, certain sizes and / or relative distances may be difficult to detect using existing sensor systems and / or difficult to perceive with the "naked eye." Furthermore, certain structures may move preoperatively (e.g., before surgery but after a preoperative scan) and / or intraoperatively. In such cases, the clinician may not be able to accurately determine the location of critical structures intraoperatively.

[0101] When the orientation of a critical structure is uncertain and / or when the proximity between a critical structure and a surgical tool is unknown, the clinician's decision-making process may be hindered. For example, a clinician may avoid certain areas in order to avoid accidentally dissecting a critical structure; however, the avoided area may be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or excessive / overly cautious operation, the clinician may be unable to enter certain desired areas. For example, excessive caution may cause the clinician to leave behind a portion of a tumor and / or other undesirable tissue in an attempt to avoid a critical structure, even if the critical structure is not in that particular area and / or will not be negatively affected by the clinician working in that particular area. In some cases, surgical results can be improved by increasing knowledge and / or certainty, which can make the surgeon more accurate in specific anatomical areas and, in some cases, make the surgeon less conservative / more aggressive.

[0102] In various aspects, the present disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities that go beyond what a clinician can see with the "naked eye" and / or beyond what an imaging system can identify and / or convey to the clinician. Various surgical visualization systems can enhance and strengthen what a clinician can know before tissue treatment (e.g., dissection) and, therefore, can improve outcomes in various situations.

[0103] For example, a visualization system may include a first light emitter configured to emit multiple spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to the spectral waves (spectral imaging), and / or the light pattern. The surgical visualization system may also include an imaging system and a control circuit in signal communication with the receiver and the imaging system. Based on the output from the receiver, the control circuit may determine a geometric surface map (i.e., a three-dimensional surface topography) of the visible surface at the surgical site and one or more distances relative to the surgical site. In some cases, the control circuit may determine one or more distances to at least partially concealed structures. Furthermore, the imaging system may communicate the geometric surface map and one or more distances to the clinician. In such cases, the enhanced view of the surgical site provided to the clinician may provide a representation of concealed structures within the relevant environment of the surgical site. For example, the imaging system may virtually enhance concealed structures on a geometric surface map of concealed and / or obstructing tissue, similar to a line drawn on the ground to indicate a practical line below the surface. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to visible obstructing tissue and / or to at least partially concealed structures and / or the depth of the concealed structures below the visible surface of the obstructing tissue. For example, the visualization system can determine the distance of an enhancement line relative to the surface of the visible tissue and communicate the distance to the imaging system.

[0104] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide valuable information to a clinician during a surgical procedure. Thus, for example, the clinician knows that the surgical visualization system is tracking critical structures that can be approached during dissection (such as ureters, specific nerves and / or critical blood vessels), and can confidently maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide instructions to the clinician for a sufficient period of time to enable the clinician to pause and / or slow down the surgical procedure and assess the proximity to the critical structure to prevent accidental damage thereto. The surgical visualization system can provide the clinician with an ideal, optimized and / or customizable amount of information to allow the clinician to confidently and / or quickly move through tissue while avoiding accidental damage to healthy tissue and / or critical structures, and thereby minimizing the risk of injury caused by the surgical procedure.

[0105] Figure 1is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In some cases, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured to identify critical structures intraoperatively and / or facilitate surgical device avoidance of critical structures 101. For example, by identifying critical structures 101, a clinician can avoid maneuvering a surgical device around a critical structure 101 and / or an area within a predetermined proximity of a critical structure 101 during a surgical procedure. For example, a clinician can avoid dissecting veins, arteries, nerves, and / or blood vessels identified as critical structures 101 and / or avoid dissecting near such critical structures. In various cases, the critical structures 101 can be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0106] The surgical visualization system 100 incorporates tissue recognition and geometric surface mapping in conjunction with the distance sensor system 104. Combined, these features of the surgical visualization system 100 can determine the position of critical structures 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system comprising, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various embodiments, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Views from the imaging device 120 can be provided to the clinician, and in various aspects of the present disclosure, these views can be enhanced with additional information based on tissue recognition, topographic mapping, and the distance sensor system 104. In such cases, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem, which can cooperate to provide clinicians with advanced data synthesis and integrated information during surgery.

[0107] The imaging device may include a camera or imaging sensor configured to detect, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). In various aspects of the present disclosure, the imaging system may include, for example, an imaging device, such as an endoscope. Additionally or alternatively, the imaging system may include, for example, an imaging device, such as an arthroscope, an angioscope, a bronchoscope, a choledochoscope, a colonoscope, a cystoscope, a duodenoscope, an enteroscope, an esophagogastroduodenoscope (gastroscope), a laryngoscope, a nasopharyngeal-nephroscope, a sigmoidoscope, a thoracoscope, a ureteroscope, or an exoscope. In other cases, such as in open surgical applications, the imaging system may not include a viewing scope.

[0108] In various aspects of the present disclosure, the tissue identification subsystem can be implemented using a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” issued on March 1, 2016, which is incorporated herein by reference in its entirety.

[0109] In various aspects of the present disclosure, the surface mapping subsystem can be implemented using a light patterning system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be used in the surgical visualization system described herein.

[0110] Structured light is the process of projecting a known pattern (usually a grid or horizontal stripes) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, are incorporated herein by reference in their entirety.

[0111] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and to determine various distances relative to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to tissue (or other structures) identified at a surgical site.

[0112] Figure 2 is a schematic diagram of a control system 133 that may be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 in signal communication with a memory 134. The memory 134 stores instructions executable by the control circuit 132 to determine and / or identify critical structures (e.g., Figure 1 101 ), determine and / or calculate one or more distances and / or three-dimensional digital representations, and transmit certain information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (e.g., Figure 1 The imaging device 120 in FIG. 1 ), one or more displays 146, or one or more controls 148, or any combination thereof. The camera 144 may include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, 3D lenses, etc.) that emit light in various visible and non-visible light spectrums. The display 146 may include one or more screens or monitors for depicting real, virtual, and / or virtual-augmented images and / or information to one or more clinicians.

[0113] In various aspects, the heart of camera 144 is the image sensor 135. Generally speaking, modern image sensors 135 are solid-state electronic devices containing up to millions of discrete photodetector sites (called pixels). Image sensor 135 technology falls into one of two categories: charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 are sensitive to wavelengths from approximately 350 nm to 1050 nm, but this range is often given as 400 nm to 1000 nm. Generally speaking, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, therefore, cannot distinguish colors. Consequently, there are two types of color CCD cameras: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution and use a mosaic (e.g., Bayer) optical filter to separate the incoming light into a series of colors and employ an interpolation algorithm to resolve the full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing a prism to direct each portion of the incoming light spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value, rather than using an algorithm to determine color. Three-chip cameras offer extremely high resolution.

[0114] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, the single light source can be pulsed to provide light in the invisible spectrum (e.g., infrared spectrum light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic component 140 can identify key structures via data from the spectral light source 150 received in part by the image sensor 135 of the camera 144. The surface mapping logic component 136 can determine the surface contour of the visible tissue based on the reflected structured light. Using the time of flight measurement results, the distance determination logic component 141 can determine one or more distances to the visible tissue and / or key structure 101. One or more outputs from the surface mapping logic 136 , tissue identification logic 140 , and distance determination logic 141 may be provided to the imaging logic 138 and may be combined, blended, and / or overlaid for communication to the clinician via a display 146 of the imaging system 142 .

[0115] The manual now briefly turns to Figures 2A to 2C , to describe various aspects of the control circuitry 132 for controlling various aspects of the surgical visualization system 100. Figure 2A , illustrates control circuitry 400 configured to control various aspects of surgical visualization system 100 in accordance with at least one aspect of the present disclosure. Control circuitry 400 may be configured to implement the various processes described herein. Control circuitry 400 may include a microcontroller comprising one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. Memory circuitry 404 stores machine-executable instructions that, when executed by processor 402, cause processor 402 to execute machine instructions to implement the various processes described herein. Processor 402 may be any of a variety of single-core or multi-core processors known in the art. Memory circuitry 404 may include volatile and non-volatile storage media. Processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from memory circuitry 404 of the present disclosure.

[0116] Figure 2B 1. Combinatorial logic circuit 410 is shown that is configured to control various aspects of surgical visualization system 100 in accordance with at least one aspect of the present disclosure. Combinatorial logic circuit 410 can be configured to implement various processes described herein. Combinatorial logic circuit 410 can include a finite state machine that includes a combinatorial logic component 412 that is configured to receive data associated with a surgical instrument or tool at an input 414, process the data via combinatorial logic component 412, and provide an output 416.

[0117] Figure 2C A sequential logic circuit 420 is shown that is configured to control various aspects of the surgical visualization system 100 in accordance with at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic component 422 may be configured to implement the various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, a combinational logic component 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 may store a current state of the finite state machine. In some cases, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic component 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinational logic component 422, and provide an output 428. In other aspects, the circuit may include a processor (e.g., Figure 2AIn other aspects, the finite state machine may include a combinational logic circuit (e.g., combinational logic circuit 410, Figure 2B ) and a combination of a sequential logic circuit 420.

[0118] See again Figure 1 In the surgical visualization system 100 of FIG. 1 , a key structure 101 can be an anatomical structure of interest. For example, the key structure 101 can be an anatomical structure such as a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor. In other cases, the key structure 101 can be, for example, a foreign structure in the anatomical field, such as a surgical device, a surgical fastener, a clamp, a tack, a bougie, a band, and / or a plate. Exemplary key structures are further described herein and in the aforementioned concurrently filed U.S. patent applications, which are incorporated herein by reference in their entirety.

[0119] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be positioned below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the critical structure 101 from the clinician's view. From the perspective of the imaging device 120, the critical structure 101 is also obscured by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other cases, the critical structure 101 can be partially obscured from view.

[0120] Figure 1 Also depicted is a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from a distal end of a shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device, such as, for example, a dissector, a stapler, a grasper, a clip applier, and / or an energy device (including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector). Additionally or alternatively, the surgical device 102 can include, for example, another imaging or diagnostic modality, such as an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to enable identification of one or more critical structures 101 and the proximity of the surgical device 102 to the critical structure 101.

[0121] The imaging device 120 of the surgical visualization system 100 is configured to be capable of detecting various wavelengths of light, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (such as a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens that are used together to simultaneously record two two-dimensional images, and thereby generate a three-dimensional image of the surgical site, render a three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to be capable of receiving images indicating the topography of visible tissue and the identification and orientation of hidden critical structures, as further described herein. For example, the field of view of the imaging device 120 may overlap with the pattern of light (structured light) on the surface 105 of the tissue, such as Figure 1 shown.

[0122] In one aspect, the surgical visualization system 100 can be incorporated into a robotic system 110. For example, the robotic system 110 can include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118, which can include servo motor controls. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. A robotic control unit can be configured to issue control motions to the robotic arms 112, 114, which can affect, for example, the surgical device 102 and the imaging device 120.

[0123] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as stripes, grid lines, and / or dots, to enable determination of the topography or terrain of the surface 105. For example, a projected light array 130 can be used for three-dimensional scanning and registration of the surface 105. The projected light array 130 can be emitted from the emitter 106 located, for example, on the surgical device 102 and / or one of the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the shape defined by the surface 105 of the tissue 103 and / or the intraoperative motion of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances relative to the surface 105.

[0124] In one aspect, the imaging device 120 may further include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable identification of the type of anatomical and / or physical structure (such as the critical structure 101). Identification of the critical structure 101 may be achieved, for example, through spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a photoacoustic imaging system. In other cases, the optical waveform emitter 123 may be located on a surgical device separate from the imaging device 120 .

[0125] The surgical visualization system 100 may also include a distance sensor system 104 that is configured to determine one or more distances at the surgical site. In one aspect, the time of flight distance sensor system 104 can be a time of flight distance sensor system that includes an emitter, such as emitter 106, and a receiver 108 that can be positioned on the surgical device 102. In other cases, the time of flight emitter can be separate from the structured light emitter. In one general aspect, the emitter 106 portion of the time of flight distance sensor system 104 can include a very tiny laser source, and the receiver 108 portion of the time of flight distance sensor system 104 can include a matching sensor. The time of flight distance sensor system 104 can detect the "time of flight" or the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. The use of a very narrow light source in the emitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104. Still referring to Figure 1 , d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 can be used to determine the transmitter-tissue distance d e Device-tissue distance d tThe device-tissue distance d can be obtained based on the known position of the transmitter 106 on the axis of the surgical device 102 relative to the distal end of the surgical device 102. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t The transmitter-tissue distance d e In some cases, the shaft of surgical device 102 may include one or more articulation joints and may be capable of articulation relative to transmitter 106 and jaws. The articulation configuration may include, for example, a multi-jointed vertebra-like structure. In some cases, a three-dimensional camera may be used to triangulate one or more distances to surface 105.

[0126] In various instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate surgical device rather than on the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In other instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate robotically controlled arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In some instances, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance from the transmitter 106 on the surgical device 102 to the surface 105 of the tissue 103 using a line between the transmitter 106 on the surgical device 102 and the imaging device 120. For example, the distance d may be determined based on the known positions of the transmitter 106 (on the surgical device 102) and the receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. e The three-dimensional position of the receiver 108 may be known and / or registered intraoperatively with the robot coordinate plane.

[0127] In some cases, the position of the transmitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other cases, the surgical visualization system 100 can be used separately from the robotic system. In such cases, the distance sensor system 104 can be independent of the robotic system.

[0128] In some cases, one or more of the robotic arms 112, 114 may be separate from the main robotic system used during the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and registered to a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or register the position of the robotic arms 112, 114 relative to a specific coordinate system. Similarly, the position of the surgical device 102 and the imaging device 120 may be registered relative to a specific coordinate system.

[0129] Still see Figure 1 , d w is the camera-to-key structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the key structure 101, and d A is the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d w The use of spectral imaging in conjunction with a time-of-flight sensor is further described herein. Furthermore, see now Figure 3 In various aspects of the present disclosure, the depth dA of the critical structure 101 relative to the surface 105 of the tissue 103 can be determined by the following method: w and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and therefore the known distance d between them). x ) to perform triangulation to determine the distance d y (It is the distance d e and d A sum).

[0130] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, the first waveform (or range of waveforms) can be used to determine the camera-critical structure distance d w , and the second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0131] Additionally or alternatively, in some cases, the distance d A It can be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance d AThis can be determined using spectral imaging because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first manner or amount, and collagen can reduce the detection signal in a different second manner or amount.

[0132] Now see Figure 4 The surgical visualization system 160 in FIG. 1 includes a surgical device 162 including an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform. The optical waveform emitter 123 can be configured to emit a waveform for determining a distance d from a common device such as the surgical device 162. t and d w , as further described herein. In such cases, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is A It can be determined as follows:

[0133] d A =d w -d t .

[0134] As disclosed herein, various information about visible tissue, embedded critical structures, and surgical devices can be determined by utilizing a combined approach that combines an image sensor configured to detect spectral wavelengths and structured light arrays with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Furthermore, the image sensor can be configured to receive visible light and, thus, provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various embodiments, the imaging device 120 may include multiple image sensors.

[0135] Camera-key structure distance d w Detection can also be performed in one or more alternative ways. In one aspect, the key structure 201 can be illuminated using, for example, fluoroscopic visualization techniques such as fluorescent indocyanine green (ICG), such as Figures 6 to 8 The camera 220 may include two optical waveform sensors 222 and 224, which simultaneously capture the left image and the right image of the key structure 201 ( Figure 7A and Figure 7B ). In such cases, the camera 220 may depict the glow of the critical structure 201 below the surface 205 of the tissue 203 and the distance d wThe distance between sensors 222 and 224 can be determined based on the known distance between them. In some cases, the distance can be more accurately determined by utilizing more than one camera or by moving the camera between multiple locations. In some aspects, one camera can be controlled by a first robotic arm, and the second camera can be controlled by another robotic arm. In such a robotic system, one camera can be, for example, a slave camera on a slave arm. The slave arm and its camera can be programmed to track the other camera and maintain, for example, a specific distance and / or lens angle.

[0136] In other aspects, the surgical visualization system 100 may employ two separate waveform receivers (ie, cameras / image sensors) to determine d w Now see Figure 9 If a critical structure 301 or its contents (eg, a vessel or vessel contents) can emit a signal 302, such as using fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0137] On the other hand, see now Figure 10A and Figure 10B The surgical visualization system may use shaking or moving the camera 440 to determine the distance d w The camera 440 is robotically controlled so that the three-dimensional coordinates of the camera 440 at different positions are known. In various cases, the camera 440 can be pivoted at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., a blood vessel or blood vessel contents) can emit a signal, such as using fluoroscopy, the actual position can be triangulated based on the camera 440 being rapidly moved between two or more known positions. Figure 10A , camera 440 is moved axially along axis A. More specifically, camera 440 is translated a distance d1 along axis A closer to key structure 401 to a position indicated as position 440', such as by moving in and out on a robotic arm. As camera 440 moves distance d1 and the size of the view changes with respect to key structure 401, the distance to key structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) can correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, camera 440 can be rotated or swept along an arc between different orientations. Referring now to Figure 10B , the camera 440 moves axially along axis A and rotates around axis A by an angle θ3. The pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. Figure 10B In FIG. 4 , the camera 440 is translated and rotated to position 440 . As the camera 440 moves and the edge of the view changes with respect to the key structure 401 , the distance to the key structure 401 can be calculated. Figure 10B, the distance d2 may be, for example, 9.01 mm, and the angle θ3 may be, for example, 0.9 degrees.

[0138] Figure 5 A surgical visualization system 500 is depicted, which is similar in many respects to surgical visualization system 100. In various embodiments, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light at multiple wavelengths to obtain, for example, spectral images of hidden structures. In various embodiments, imaging device 520 can also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown being used intraoperatively to identify and facilitate avoidance of certain critical structures that are not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (in this example, the uterus).

[0139] The surgical visualization system 500 is configured to be able to determine an emitter-tissue distance d from an emitter 506 on a surgical device 502 to a surface 505 of a uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to detect the presence of a transducer based on the transmitter-tissue distance d e To extrapolate the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 t The surgical visualization system 500 is further configured to determine the tissue-ureter distance d from the ureter 501a to the surface 505. A and the camera-ureter distance d from the imaging device 520 to the ureter 501a w As this article about Figure 1 As described above, for example, the surgical visualization system 500 can utilize, for example, spectral imaging and a time-of-flight sensor to determine the distance d w In various circumstances, the surgical visualization system 500 can determine (eg, triangulate) the tissue-ureter distance d based on other distance and / or surface mapping logic described herein. A (or depth).

[0140] Now see Figure 11, which depicts a schematic diagram of a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For example, control system 600 is a conversion system that integrates spectral signature tissue recognition and structured light tissue localization to identify key structures, particularly when these structures are obscured by other tissues (such as fat, connective tissue, blood, and / or other organs). Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue from healthy tissue within an organ.

[0141] The control system 600 is configured to implement a hyperspectral imaging and visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field of view. The control system 600 includes conversion logic 648 to convert tissue data into information usable by the surgeon. For example, variable reflectivity based on wavelength relative to an obscuring material can be used to identify critical structures within the anatomy. Furthermore, the control system 600 combines the identified spectral signatures with structured light data within an image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical applications in a system with enhanced image overlays. The technology can be used both intraoperatively and preoperatively with the additional visual information. In various scenarios, the control system 600 is configured to provide a warning to the clinician when approaching one or more critical structures. Various algorithms can be employed to guide robotic automation and semi-automated approaches based on the surgical procedure and proximity to critical structures.

[0142] Projected light arrays are used to determine tissue shape and motion intraoperatively. Alternatively, flash lidar can be used for surface mapping of tissue.

[0143] The control system 600 is configured to detect critical structures and provide image overlays of the critical structures, and to measure distances to the surface of visible tissue and distances to embedded / buried critical structures. In other cases, the control system 600 may measure distances to the surface of visible tissue or detect critical structures and provide image overlays of the critical structures.

[0144] The control system 600 includes a spectrum control circuit 602. For example, the spectrum control circuit 602 may be a field programmable gate array (FPGA) or a Figures 2A to 2CAnother suitable circuit configuration is described. Spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. For example, processor 604 can be configured for hyperspectral processing and can utilize C / C++ code. For example, video input processor 606 receives video input including control (metadata) data, such as shutter time, wavelength, and sensor analysis. Processor 604 is configured to process the video input signal from video input processor 606 and provide a video output signal to video output processor 608, which includes, for example, a hyperspectral video output that interfaces with control (metadata) data. Video output processor 608 provides the video output signal to image overlay controller 610.

[0145] The video input processor 606 is coupled to a camera 612 at the patient side via a patient isolation circuit 614. As previously described, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers to isolate the patient from other circuits in the system. The camera 612 receives intraoperative images through optics 632 and an image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include, for example, a CMOS image sensor. Figure 2 Any image sensor technology described herein. In one aspect, camera 612 outputs images at a 14-bit / pixel signal. It should be understood that higher or lower pixel resolutions may be employed without departing from the scope of this disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fused output signal is provided to a video input processor 606 and a laser pulse control circuit 622.

[0146] The laser pulse control circuit 622 controls the laser engine 624. The laser engine 624 outputs a plurality of wavelengths (λ1, λ2, λ3, ..., λ n ) light. Laser engine 624 can operate in multiple modes. In one aspect, laser engine 624 can operate in two modes, for example. In a first mode (e.g., normal operating mode), laser engine 624 outputs an illumination signal. In a second mode (e.g., recognition mode), laser engine 624 outputs RGBG and NIR light. In various cases, laser engine 624 can operate in a polarization mode.

[0147] Light output 626 from the laser engine 624 illuminates the target anatomical structure in the intraoperative surgical site 627. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630, which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) onto the surgical tissue or organ at the surgical site 627. The camera 612 receives the patterned light and the reflected light output by the camera optics 632. The image sensor 634 converts the received light into a digital signal.

[0148] The color RGB fusion circuit 616 also outputs a signal to the image overlay controller 610 and the video input module 636 for reading the laser pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0149] The first video output signal 640 and the second video output signal 642 include data representing the location of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. In combination with the data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine the distance d to the buried critical structure. A ( Figure 1 ) (e.g., via triangulation algorithm 644), and the distance d A The image overlay controller 610 may be provided via the video output processor 646. The conversion logic components described above may include the conversion logic circuit 648, the intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0150] In various cases, preoperative data 650 from a CT or MRI scan can be used to register or match certain three-dimensional deformable tissues. Such preoperative data 650 can be provided to an integration module 643 and ultimately to an image overlay controller 610 so that such information can be overlaid with the view from the camera 612 and provided to a video monitor 652. Registration of preoperative data is further described herein and in the aforementioned concurrently filed U.S. patent applications (including, for example, U.S. patent application Ser. No. 16 / 128,195, entitled "INTEGRATION OF IMAGING DATA"), which are incorporated herein by reference in their entirety.

[0151] The video monitor 652 can output the integrated / enhanced view from the image overlay controller 610. The clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view in which a three-dimensional rendering of visible tissue is depicted and (B) an enhanced view in which one or more hidden critical structures are depicted on the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can, for example, switch distance measurements to the surface of one or more hidden critical structures and / or visible tissue.

[0152] The control system 600 and / or its various control circuits may be incorporated into the various surgical visualization systems disclosed herein.

[0153] Figure 12 7. A structured (or patterned) light system 700 is shown in accordance with at least one aspect of the present disclosure. As described herein, structured light in the form of stripes or lines, for example, can be projected from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. Figure 1 ) can be configured to detect a projected pattern of light on surface 705. The way in which the projected pattern deforms when it strikes surface 705 allows the vision system to calculate depth and surface information of the target anatomy.

[0154] In some cases, invisible (or imperceptible) structured light can be used without interfering with other computer vision tasks where the projected pattern might confuse the image. For example, infrared light that alternates between two diametrically opposed patterns or extremely fast visible light frame rates can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0155] Now see Figure 13, illustrating the concept of hyperspectral imaging by way of example, shows a terrestrial hyperspectral imaging system 800. For example, the terrestrial hyperspectral imaging system 800 is configured to image terrestrial features or objects, such as soil, water, and / or vegetation. The terrestrial hyperspectral imaging system 700 includes a spaceborne hyperspectral sensor 822 on a spacecraft 820 to hyperspectrally image a portion of the Earth's surface 805. The spectral dimension includes several layers. Each pixel of the image contains a sampled spectrum that is used to identify the materials present in the pixel by their reflectivity. The data can be converted into graphical representations 850, 852, 854 of reflectivity as a function of wavelength for soil, water, and vegetation, respectively. Terrestrial hyperspectral imaging is further described at www.markelowitz.com / Hyperspectral.html.

[0156] The concept of hyperspectral imaging is also shown by way of example. Figure 14 850 is a graphical representation of hyperspectral signatures of various terrestrial features or objects according to at least one aspect of the present disclosure. Reflectance percentage is plotted along the vertical axis, while wavelength (nm) is plotted along the horizontal axis. As shown, each object (pine forest, grassland, red sand pit, and muddy water) has a unique hyperspectral signature that can be used to identify the object.

[0157] According to at least one aspect of the present disclosure, Figure 13 and Figure 14 The hyperspectral imaging concept described can be used for different materials with different absorption wavelengths and bands. The following table shows the absorption wavelengths and bands of various materials. The first wavelength range between 400nm and 700nm represents the visible light spectrum. The second wavelength range between 700nm and 1400nm represents the near-infrared (NIR) spectrum. The third wavelength range between 1400nm and 3000nm represents the short-wave infrared (SWIR) spectrum. The first band centered at 1250nm represents iron absorption and leaf water content. The second band between 1500nm and 1750nm represents plastics, fiberglass, and oil. The third band between 200nm and 2400nm represents mineral ID.

[0158] Table 1 specifies the absorption wavelengths and bands for various materials.

[0159]

[0160] Table 1

[0161] Now see Figures 15A to 15CAs a further illustration of the concept of hyperspectral imaging, a test was conducted in which spectral imaging was applied to a fried egg 952. An image of the fried egg 952 having a yellow yolk 954 and egg white 956 surrounding the yolk 954 was obtained. Figure 15A A graphical representation 950 of the spectral signature of a fried egg 952 is shown in FIG. Figure 15B Specifically, graphical representation 950 shows the absorbance units of yolk 954 and egg white 956 of a fried egg 952 versus wavelength (nm). Figure 15C , a spectral image of a fried egg 952 (shown in black and white) is shown, where the image is enhanced to distinguish between the yolk and egg white portions based on hyperspectral feature data.

[0162] In various situations, hyperspectral imaging techniques, as described herein for illustrative purposes with respect to land features and objects and fried eggs, can be employed to identify features in anatomical structures in order to distinguish critical structures from obscurants. For example, hyperspectral imaging techniques can provide a visualization system that can provide a means of identifying critical structures such as ureters and / or blood vessels, particularly when those structures are obscured by, for example, fat, connective tissue, blood, or other organs. Differences in reflectivity at different wavelengths in the infrared (IR) spectrum can be used to determine the presence of critical structures versus obscurants. See now Figures 16 to 18 , depicting exemplary hyperspectral signatures of, for example, ureters, arteries, and neural tissue relative to obscurants such as fat, lung tissue, and blood.

[0163] Figure 16 is a graphical representation of exemplary ureteral features relative to an obscuration 1050. The graph shows the reflectivity of fat, lung tissue, blood, and ureter versus wavelength (nm) for each wavelength. Figure 17 is a graphical representation of exemplary arterial features relative to obscuration 1052. The graph shows the reflectivity of fat, lung tissue, blood, and blood vessels versus wavelength (nm). Figure 18 is a graphical representation of exemplary nerve features relative to obscurants 1054. The graph shows the reflectivity of fat, lung tissue, blood, and nerves versus wavelength (nm).

[0164] In various cases, selected wavelengths for spectral imaging (i.e., "selective spectral" imaging) can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site. By utilizing selective spectral imaging, the amount of time required to obtain spectral images can be minimized, allowing information to be obtained in real time or near real time and utilized intraoperatively. In various cases, the wavelength can be selected by the clinician or by the control circuit based on the clinician's input. In some cases, the wavelength can be selected based on, for example, machine learning and / or big data that the control circuit can access via the cloud.

[0165] The aforementioned application of spectral imaging to tissue can be used intraoperatively to measure the distance between a waveform emitter and a critical structure obscured by tissue. In one aspect of the present disclosure, referring now to Figure 19 and Figure 20 , showing a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125. In some cases, the time-of-flight sensor system 1104 may be incorporated into the surgical visualization system 100 ( Figure 1 ). Time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. Transmitted wave 1124 extends from transmitter 1106 to critical structure 1101, and received wave 1125 is reflected back from critical structure 1101 by receiver 1108. Surgical device 1102 is positioned through a trocar 1110 that extends into a cavity 1107 of a patient.

[0166] The waveforms 1124, 1125 are configured to be able to penetrate the obscured tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or a photoacoustic signal may be emitted from the emitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, the waveforms 1124, 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, the emitter 1106 is configured to be able to provide a binary signal on and off, such as Figure 20 As shown, for example, the binary signal may be measured by receiver 1108 .

[0167] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to be able to determine the distance d ( Figure 19 ). Figure 19 The time-of-flight timing diagram 1130 of the transmitter 1106 and receiver 1108 is shown in Figure 20 The delay is a function of the distance d, and the distance d is given by:

[0168]

[0169] in:

[0170] c = speed of light;

[0171] t = length of the pulse;

[0172] q1 = charge accumulated when light is emitted; and

[0173] q2 = Charge accumulated when no light is emitted.

[0174] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to Figure 19 In various cases, additional transmitter / receivers and / or pulsed signals from transmitter 1106 can be configured to transmit non-penetrating signals. Non-tissue penetrating signals can be configured to determine the distance from the transmitter to the surface 1105 of the obscuring tissue 1103. In various cases, the depth of the critical structure 1101 can be determined by the following formula:

[0175] d A =d w -d t .

[0176] in:

[0177] d A = depth of key structure 1101;

[0178] d w = distance from transmitter 1106 to key structure 1101 ( Figure 19 d) in the above; and

[0179] d t , = the distance from the emitter 1106 (on the distal end of the surgical device 1102 ) to the surface 1105 of the shielding tissue 1103 .

[0180] In one aspect of the present disclosure, see now Figure 21 , shows a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c. In some cases, the time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 ( Figure 1 ). The time-of-flight sensor system 1204 includes a waveform transmitter 1206 and a waveform receiver 1208. The waveform transmitter 1206 is positioned on the first surgical device 1202a, and the waveform receiver 1208 is positioned on the second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their respective trocars 1210a and 1210b, respectively, which extend into a cavity 1207 of a patient. Transmitted waves 1224a, 1224b, and 1224c extend from the transmitter 1206 toward the surgical site, and received waves 1225a, 1225b, and 1225c are reflected from various structures and / or surfaces at the surgical site back to the receiver 1208.

[0181] Different emission waves 1224a, 1224b, 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c can be in the visible light, NIR, or SWIR wavelength spectrum. For example, visible light can be reflected from the surface 1205 of the tissue 1203, and NIR and / or SWIR waveforms can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, as described herein, a spectral signal (e.g., a hyperspectral, multispectral, or selective spectral) or a photoacoustic signal can be emitted from the emitter 1206. In various cases, waves 1224b, 1224c can be selected to target critical structures 1201a, 1201b within tissue 1203 based on spectral characteristics of the critical structures 1201a, 1201b, as further described herein. Photoacoustic imaging is further described herein and in the aforementioned concurrently filed U.S. patent applications, which are incorporated herein by reference in their entirety.

[0182] The transmitted waves 1224a, 1224b, 1224c may be reflected from the target material (ie, the surface 1205, the first critical structure 1201a, and the second critical structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c may be due to Figure 21 The distance d shown 1a d 2a d 3a d 1b d 2b d 2c and was delayed.

[0183] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 are independently positionable (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d may be calculated based on the known positions of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when the surgical devices 1202a, 1202b are robotically controlled, these positions may be known. Knowledge of the positions of the emitter 1206 and receiver 1208 and the time at which the photon stream is targeted at a certain tissue and the information of the specific response received by the receiver 1208 may allow the distance d to be determined. 1a d 2a d3a d 1b d 2b d 2c In one aspect, the distance to the obscured critical structures 1201a, 1201b can be triangulated using the penetration wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0184] Still see Figure 21 In various cases, the receiver 1208 can be rotated so that the center of mass of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b, in the view provided to the clinician. Such an orientation can quickly convey one or more relevant distances and / or viewing angles with respect to the critical structure. For example, Figure 21 As shown, the surgical site is displayed from a perspective of key structures 1201a perpendicular to the viewing plane (i.e., with the vessels oriented in-page / out-of-page). In various cases, such an orientation may be the default setting; however, the view may be rotated or otherwise adjusted by the clinician. In some cases, the clinician may switch between different surfaces and / or target structures that define the perspective of the surgical site provided by the imaging system.

[0185] In various cases, the receiver 1208 can be mounted on a trocar or cannula (such as trocar 1210b), for example, through which the surgical device 1202b is positioned. In other cases, the receiver 1208 can be mounted on a separate robotic arm whose three-dimensional position is known. In various cases, the receiver 1208 can be mounted on a movable arm separate from the robot that controls the surgical device 1202a, or can be mounted to an operating room (OR) table that can be registered with the robot's coordinate plane during surgery. In such cases, the positions of the transmitter 1206 and the receiver 1208 can be registered with the same coordinate plane, so that distances can be triangulated based on the output from the time-of-flight sensor system 1204.

[0186] Combining a time-of-flight sensor system and near-infrared spectroscopy (NIRS), known as TOF-NIRS, which is capable of measuring time-resolved signatures of NIR light with nanosecond resolution, is described in an article entitled “TIME-OF-FLIGHT NEAR-INFRAREDSPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY INGRAPEFRUIT” (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228), which is incorporated herein by reference in its entirety and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0187] In various cases, the time-of-flight spectral waveform is configured to determine the depth of a critical structure and / or the proximity of a surgical device to a critical structure. In addition, various surgical visualization systems disclosed herein include a surface mapping logic component that is configured to create a three-dimensional rendering of the surface of visible tissue. In such cases, the clinician can be aware of the proximity (or lack thereof) of the surgical device to the critical structure even when visible tissue obstructs the critical structure. In one case, the topography of the surgical site is provided on a monitor by the surface mapping logic component. If the critical structure is close to the surface of the tissue, spectral imaging can convey the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on the known limitations of the spectral imaging system, the system is configured to convey that the critical structure is out of range even if the spectral imaging system cannot detect the critical structure at all. Therefore, the clinician can continue to move the surgical device and / or manipulate the tissue. When the critical structure moves into the range of the spectral imaging system, the system can identify the structure and therefore convey that the structure is within range. In such cases, an alert can be provided when a structure is initially identified and / or further moved within a predefined proximity zone. In such cases, proximity information (i.e., lack of proximity) can be provided to the clinician even if the spectral imaging system does not identify a critical structure using known boundaries / ranges.

[0188] Various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and alert the clinician before damaging a critical structure through accidental dissection and / or transection. In various aspects, the surgical visualization system is configured to identify, for example, one or more of the following critical structures: ureters, intestines, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve and their branches), blood vessels (including the pulmonary artery and lobar arteries and pulmonary veins and lobar veins, the inferior mesenteric artery [IMA] and its branches, the superior rectal artery, the sigmoid artery and the left colic artery), the superior mesenteric artery (SMA) and its branches (including the middle colic artery, the right colic artery, the ileal artery), the hepatic artery and its branches, the portal vein and its branches, the splenic artery / vein and its branches, the external and internal (lower abdominal) ileal vessels, the short gastric arteries, the uterine arteries, the middle sacral vessels and lymph nodes. In addition, the surgical visualization system is configured to indicate the proximity of a surgical device to a critical structure and / or alert the clinician when a surgical device is in proximity to a critical structure.

[0189] Various aspects of the present disclosure provide intraoperative critical structure identification (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, which enable visualization of critical structures, for example, below the surface of tissue (such as 1.0 cm to 1.5 cm below the surface of the tissue). In other cases, the surgical visualization system may identify structures less than 1.0 cm or greater than 1.5 cm below the surface of the tissue. For example, if the structure is not visible due to depth, even a surgical visualization system that can identify structures only within 0.2 mm of the surface may be valuable. In various aspects, the surgical visualization system may enhance the clinician's view, for example, using a virtual depiction of the critical structure as a visible white light image superimposed on the surface of the visible tissue. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and may, for example, provide a proximity alert when the distal tip of the surgical instrument moves within a specific range of the critical structure (such as within 1.0 cm of the critical structure).

[0190] Various surgical visualization systems disclosed herein can identify when dissection is too close to a critical structure. Dissection may be "too close" to a critical structure based on temperature (i.e., too much heat near a critical structure that could risk damaging / heating / melting the critical structure) and / or tension (i.e., too much tension near a critical structure that could risk damaging / tearing / pulling the critical structure). For example, such surgical visualization systems can facilitate dissection around a vessel when skeletonizing the vessel prior to ligation. In various cases, a thermal imaging camera can be utilized to read the heat at the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the tool temperature is above a predefined threshold (such as, for example, 120°F), a warning can be provided to the clinician at a first distance (such as, for example, 10 mm), and if the tool temperature is less than or equal to the predefined threshold, a warning can be provided to the clinician at a second distance (such as, for example, 5 mm). The predefined thresholds and / or warning distances can be default settings and / or programmable by the clinician. Additionally or alternatively, the proximity alert may be associated with a thermal measurement made by the tool itself, such as a thermocouple measuring heat in the distal jaws of, for example, a monopolar or bipolar dissector or vessel sealer.

[0191] The various surgical visualization systems disclosed herein can provide sufficient sensitivity with respect to critical structures and specificity to enable clinicians to confidently perform rapid but safe dissections based on standards of care and / or device safety data. The systems can function intraoperatively and in real time during surgical procedures with minimal, and in various cases, no, ionizing radiation risk to the patient or clinician. In contrast, during fluoroscopy, the patient and clinician can be exposed to ionizing radiation via, for example, an X-ray beam used to view anatomical structures in real time.

[0192] The various surgical visualization systems disclosed herein can be configured to, for example, detect and identify one or more desired types of critical structures in a forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more desired types of critical structures, for example, in the surrounding area of ​​the surgical device and / or in multiple planes / dimensions.

[0193] The various surgical visualization systems disclosed herein can be easy to operate and / or interpret. Furthermore, the various surgical visualization systems can incorporate an "override" feature that allows a clinician to override default settings and / or operations. For example, a clinician can selectively disable alerts from the surgical visualization system and / or move closer to a critical structure than the surgical visualization system suggests, such as when the risk to the critical structure is less than the risk of avoiding the area (e.g., when removing cancer surrounding a critical structure, the risk of leaving cancerous tissue behind may be greater than the risk of causing damage to the critical structure).

[0194] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, the specific implementation of a surgical visualization system may not change the way surgical procedures are performed. Furthermore, a surgical visualization system may be more economical compared to the cost of accidental transections. Data suggests that reduced accidental damage to critical structures can drive incremental compensation.

[0195] The various surgical visualization systems disclosed herein can operate in real time or near real time and far enough in advance to enable the clinician to anticipate critical structures. For example, the surgical visualization system can provide sufficient time to "slow down, assess, and avoid" in order to maximize the efficiency of the surgical procedure.

[0196] The various surgical visualization systems disclosed herein may not require contrast agents or dyes to be injected into tissue. For example, spectral imaging can be configured to visualize hidden structures intraoperatively without the use of contrast agents or dyes. In other cases, contrast agents can be more easily injected into the appropriate tissue layers than with other visualization systems. For example, the time between contrast agent injection and visualization of critical structures can be less than two hours.

[0197] The various surgical visualization systems disclosed herein can be associated with clinical data and / or device data. For example, the data can provide boundaries regarding how close an energy-enabled surgical device (or other potentially damaging device) should be to tissue that the surgeon does not want to damage. Any data modules that interact with the surgical visualization systems disclosed herein can be provided integrally with or separately from a robot to enable use with standalone surgical devices, such as in open or laparoscopic surgery. In various cases, the surgical visualization system can be compatible with a robotic surgical system. For example, visualization images / information can be displayed on a robotic console.

[0198] In various cases, the clinician may not know the position of the critical structure relative to the surgical tool. For example, when the critical structure is embedded in the tissue, the clinician may not be able to determine the position of the critical structure. In some cases, the clinician may want to keep the surgical device outside the range of the orientation around the critical structure and / or away from the visible tissue covering the hidden critical structure. When the orientation of the hidden critical structure is unknown, the clinician may have the risk of moving too close to the critical structure, which may cause unintentional trauma to the critical structure and / or dissection and / or excessive energy, heat and / or tension near the critical structure. Alternatively, the clinician may keep too far away from the suspected position of the critical structure and have the risk of affecting the tissue in an attempt to avoid the critical structure at a less ideal position.

[0199] The present invention provides a surgical visualization system that presents tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of a surgical device relative to a critical structure. Such tracking can occur intraoperatively, in real time, and / or near real time. In various cases, the tracking data can be provided to a clinician via a display screen (e.g., a monitor) of an imaging system.

[0200] In one aspect of the present disclosure, a surgical visualization system includes: a surgical device including an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect the structured light pattern on an embedded structure and the visible surface; and control circuitry in signal communication with the camera and the imaging system, wherein the control circuitry is configured to determine a distance from the surgical device to the embedded structure and provide a signal indicating the distance to the imaging system. For example, the distance can be determined by calculating the distance from the camera to a critical structure illuminated using fluoroscopy and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses of the camera (e.g., a left lens and a right lens). For example, the distance from the surgical device to the critical structure can be triangulated based on the known positions of the surgical device and the camera. Alternative devices for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue that overlays / covers the embedded critical structure. For example, the surgical visualization system can identify and enhance the view of hidden critical structures by drawing a schematic representation of the hidden critical structures on the visible structures (such as a line on the surface of the visible tissue). The surgical visualization system can also determine the distance to the enhanced line on the visible tissue.

[0201] By providing clinicians with up-to-date information about the proximity of surgical devices to hidden critical structures and / or visible structures, as provided by various surgical visualization systems disclosed herein, clinicians can make more informed decisions regarding the placement of surgical devices relative to hidden critical structures. For example, clinicians can view the distance between surgical devices and critical structures in real time / intraoperatively, and in some cases, alerts and / or warnings can be provided by the imaging system when the surgical device moves within a predefined proximity and / or zone of a critical structure. In some cases, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly" zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more of the critical structure). In such cases, clinicians can maintain momentum throughout the surgical procedure without having to monitor the suspected location of critical structures and the proximity of the surgical device thereto. As a result, certain surgical procedures can be performed faster, with fewer pauses / interruptions, and / or with improved accuracy and / or certainty. In one aspect, surgical visualization systems can be used to detect tissue variability, such as variability of tissue within an organ, to differentiate between tumors / cancer tissue / unhealthy tissue and healthy tissue. Such surgical visualization systems can maximize the removal of unhealthy tissue while minimizing the removal of healthy tissue.

[0202] Now see Figures 22 to 24 , shows a surgical visualization system 1400 including a three-dimensional camera 1420 and a surgical device 1402. The camera 1420 includes an image sensor, as further described herein. In various aspects, the surgical visualization system 1400 can be similar to the surgical visualization system 100 ( Figure 1 For example, the surgical visualization system 1400 may be configured to identify one or more critical structures 1401 embedded beneath a surface 1405 of the tissue 1403 and determine one or more distances relative to the surface 1405 and / or the critical structures 1401 .

[0203] Surgical device 1402 includes an emitter 1406 configured to emit a pattern of structured light. The pattern is configured to reflect from a surface, and the reflected light can be detected by a camera or sensor (such as camera 1420). Based on the reflected pattern, the control circuit is configured to determine the contour of surface 1405 and one or more distances relative to surface 1405. For example, the control circuit can generate a three-dimensional model of the surface. Various control circuits are further described herein (e.g., see Figure 2 and Figure 11 ).

[0204] Camera 1420 is a three-dimensional camera. For example, camera 1420 includes a two-dimensional left lens or sensor 1408a and a two-dimensional right lens or sensor 1408b. Lenses 1408a, 1408b are configured to detect one or more key structures (such as key structure 1401) embedded in tissue 1403. In various cases, key structure 1401 can be illuminated with a contrast agent such as ICG and imaged via, for example, fluoroscopy. In addition, left lens 1408a is configured to detect a two-dimensional view (such as a right lens) of the illuminated key structure 1401. Figure 23 ), and the right lens 1408b is configured to detect another two-dimensional view of the illuminated key structure 1401 (also shown Figure 23 shown).

[0205] The two-dimensional views from lenses 1408a, 1408b can be integrated or combined to generate a three-dimensional view of the illuminated critical structure 1401 embedded in the anatomical target tissue. In various cases, the distance from the surgical device 1402 to the critical structure 1401 can be determined from the image obtained by camera 1420. For example, the distance can be determined by applying an algorithm to the two images obtained by lenses 1408a, 1408b. For example, the left hand image ( Figure 23 ) in the distance d l and the right-hand image ( Figure 23 ) in the distance d r can be averaged to obtain the distance d from the surgical device 1402 to the critical structure 1401 ( Figure 22 ), as follows:

[0206]

[0207] In various cases, because the relative positions of the camera 1420 and the surgical device 1402 are known (i.e., in a registered coordinate system, such as the same robot coordinate system), for example, when the distance from the critical structure 1401 to the camera 1420 is known, the distance from the surgical device 1402 to the critical structure 1401 can be triangulated. The robot has knowledge of the coordinate positions of the arm (e.g., the surgical device 1402 and the camera 1420) and can therefore perform triangulation to determine the intersection distance. Alternative means for determining the distance d between the surgical device and the critical structure (e.g., ultrasound, time-of-flight measurement systems, hyperspectral signal analysis, etc.) are further described herein.

[0208] Main references Figure 23, a three-dimensional image may be provided to a display screen 1450, such as a video monitor of an imaging system. For example, the display screen 1450 may selectively depict a three-dimensional real-time image / video feed obtained by the camera 1420. In various cases, the display screen 1450 may also selectively depict a three-dimensional rendering of the surgical site (e.g., visible tissue) based on the surface mapping logic and the structured light pattern emitted from the emitter 1406 and detected by the camera 1420. In some cases, the three-dimensional rendering of the visible tissue at the surgical site may be superimposed, for example, with the embedded critical structure 1401. Additionally or alternatively, the distance d ( Figure 22 ) can be displayed on display screen 1450.

[0209] In one case, still see Figure 23 , the display screen 1450 may be configured to display an enhanced color schematic 1452a of the key structure 1401 having a color legend or key 1454a to indicate the distance d to the key structure 1401 and / or portions thereof. Figure 22 ). For example, key 1454a provides a color spectrum corresponding to linear distance or area, such as green, where the surgical device 1402 is 15 mm to 20 mm from the critical structure 1401; yellow, where the surgical device 1402 is 15 mm to 10 mm from the critical structure 1401; orange, where the surgical device 1402 is 5 mm to 10 mm from the critical structure 1401; and red, where the surgical device 1402 is 0 mm to 5 mm from the critical structure 1401. Different colors can correspond to different types of communication with the clinician. For example, green can mean "continue" or "proceed," yellow can mean "slow down" or "proceed with caution," orange can mean "proceed with extreme caution," and red can mean "stop." In such cases, the critical structure can be depicted with one or more colors corresponding to the calculated distance and an associated color on key 1454a depending on the proximity of the surgical device 1402 to the portion of the critical structure 1401.

[0210] In various instances, the clinician can be alerted when the distance moves from one proximity zone to another along the spectrum of key 1454a. For example, a first alert can be provided when the distance moves from the green zone to the yellow zone, a second alert can be provided when the distance moves from the yellow zone to the orange zone, and a third alert can be provided when the distance moves from the orange zone to the red zone. The type of alert can vary depending on the proximity of the surgical device 1402 and / or the preprogrammed settings selected by the clinician. Similarly, in various aspects, the imaging system can alert the clinician when the surgical device 1402 moves away from the critical structure 1401, such as when the critical structure 1401 is the target of the surgical device 1402.

[0211] In another case, see Figure 24 , the display screen 1450 may be configured to display a black and white cross-hatched enhanced schematic diagram 1452b of the key structure 1401 with a cross-hatched legend or key 1454b to indicate the distance d ( Figure 22 ).

[0212] In some cases, the distance d from the surgical device 1402 to the critical structure 1401 may be displayed along a proximity spectrum indicator that includes multiple ranges and / or regions corresponding to different distances. Different types of cross-hatching may correspond to different proximity regions, and a key 1454b may explain the importance of the different types of cross-hatching relative to the distances or proximity regions. For example, the proximity spectrum may be defined by a color spectrum, a range of numerical values, and / or other distinguishing symbols indicating proximity.

[0213] In various situations, the surgical visualization systems disclosed herein (such as surgical visualization system 1400) can be used to target specific critical structures or portions thereof. For example, see again Figure 22 , a critical structure 1401 can be a structure targeted by a surgical device 1402. In one instance, the critical structure 1401 can be a location on a blood vessel that a clinician desires to dissect in order to remove a piece of tissue without impairing blood flow to other tissues. For example, the critical structure 1401 can be a location on a blood vessel downstream of a branch that supplies blood to healthy tissue. In such instances, the downstream location on the blood vessel can be marked and targeted by the surgical visualization system. Marking of structures is further described in concurrently filed U.S. patent application Ser. No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MONITORING,” which is hereby incorporated by reference in its entirety. The proximity spectral indicator can be configured to track / monitor the proximity of the surgical device 1402 to the targeted critical structure 1401.

[0214] Now see Figure 25 and Figure 26 , shows a surgical visualization system 1500 including a camera 1520 and surgical devices 1502a, 1502b, 1502c. Camera 1520 includes an image sensor, as further described herein. Surgical visualization system 1500 can be similar in various respects to surgical visualization system 100 ( Figure 1 ) and surgical visualization system 1400 ( Figures 22 to 24 For example, the surgical visualization system 1500 may be configured to identify one or more critical structures 1501 embedded beneath a surface 1505 of the tissue 1503 and determine one or more distances relative to the critical structures 1501 .

[0215] Similar to surgical device 1402 ( Figure 22 ), one or more of the surgical devices 1502a, 1502b, 1502c may include an emitter configured to emit light waves. For example, the emitter may be configured to emit a pattern of structured light. The pattern is configured to reflect from a surface, and the reflected light may be detected by a camera or sensor (such as camera 1520). Based on the reflected pattern, the control circuit is configured to determine the contour of the surface 1505 and various distances relative to the surface 1505. For example, the control circuit may generate a three-dimensional model of the surface 1505. Various control circuits are further described herein (e.g., see Figure 2 and Figure 11 ).

[0216] Similar to camera 1420 ( Figure 22 ), camera 1520 is a three-dimensional camera that includes a two-dimensional left lens or sensor 1508a and a two-dimensional right lens or sensor 1508b. Lenses 1508a, 1508b are configured to detect embedded key structure 1501. For example, left lens 1508a is configured to detect a two-dimensional view of key structure 1501, and right lens 1508b is configured to detect another two-dimensional view of key structure 1501. The two-dimensional views obtained by lenses 1508a, 1508b can be combined to display a display screen 1550 ( Figure 26 ) generates a three-dimensional view 1552 of the surgical site on the display screen 1550. The display screen 1550 may be a video monitor that provides a video feed of the surgical site and / or various additional data / information to the clinician during surgery. For example, the three-dimensional view 1552 on the display screen 1550 may depict the relative positions of the key structures 1501 and surgical devices 1502a, 1502b, 1502c, with an overlay of a topographical surface map of the tissue 1503 from the structured light and surface mapping logic.

[0217] Furthermore, in various cases, the distances d1, d2, and d3 from the surgical devices 1502a, 1502b, 1502c, respectively, to the critical structure 1501 can be determined and / or estimated based on three-dimensional imaging of the surgical site, as further described herein. For example, the distance from each surgical device 1502a, 1502b, 1502c to the critical structure 1501 can be measured in two dimensions and averaged, or otherwise calculated / triangulated from known positions and / or distances.

[0218] The distances d1, d2, and d3 are transmitted to a display screen 1550 (e.g., a video monitor), which is configured to display the distances d1, d2, and d3. For example, the distances d1, d2, and d3 may be displayed in a color-coded format, such as Figure 26More specifically, distances d1, d2, and d3 may be displayed along proximity spectrum indicators 1554a, 1554b, and 1554c, respectively, that indicate the distances. The proximity spectrum indicators 1554a, 1554b, and 1554c include a spectrum of colors arranged along a strip that may correspond to an area surrounding a critical structure. Figure 26 , the green area corresponds to a distance of 15 mm to 20 mm, the yellow area corresponds to a distance of 15 mm to 10 mm, the orange area corresponds to a distance of 5 mm to 10 mm, and the red area corresponds to a distance of 0 mm to 5 mm. The proximity areas can be indicated along the proximity spectrum by markings (such as arrows 1556 a, 1556 b, 1556 c) indicating the areas of each surgical device 1502 a, 1502 b, 1502 c, respectively.

[0219] Additionally or alternatively, the proximity spectrum indicators 1554a, 1554b, 1554c may include a series of numerical values ​​and / or other symbols indicating proximity zones / distances. In some cases, proximity zones may be configured or selected for different critical structures and / or different surgical devices. For example, the proximity zones around blood vessels may be different for energy devices and staplers. Furthermore, the proximity zones for energy devices may be different for veins and arteries.

[0220] In various circumstances, the surgical visualization system 1500 is configured to provide an alert, warning, or other indication to the clinician when one of the surgical devices 1502a, 1502b, 1502c approaches a predefined position range / minimum distance limit / proximity zone around the critical structure 1501. The alert may be provided at different critical distances based on the type of surgical device 1502a, 1502b, 1502c and the type of critical structure 1501.

[0221] In various cases, one or more of the surgical devices 1502a, 1502b, 1502c may be robotic tools. For example, a robotic system may control the surgical devices 1502a, 1502b, 1502c. In some cases, proximity zones around critical structures may create "stop zones," and the robotic system may automatically control the surgical devices 1502a, 1502b, 1502c to keep them outside of the "stop zones." If a clinician provides input commands to move the surgical devices 1502a, 1502b, 1502c into the "stop zones," the automated tool control motion may prevent the surgical devices 1502a, 1502b, 1502c from entering the defined "stop zones." In some cases, such automated tool control may, for example, be toggled on / off, set as a default, and / or be overridden by a clinician's overriding input.

[0222] Now see Figures 27 to 31 , a surgical visualization system 1600 is shown. In various aspects, the surgical visualization system 1600 can be similar to the surgical visualization system 100 ( Figure 1 ). For example, the surgical visualization system 1600 can be configured to identify one or more critical structures 1601 embedded below the surface 1605 of the tissue 1603 and determine one or more distances relative to the surface 1605 and / or the critical structures 1601. The surgical visualization system 1600 includes a detector or camera 1620 and a surgical device 1602. The camera 1620 includes an image sensor, as further described herein. The surgical visualization system 1600 is configured to provide a visible light reproduction of the surgical field in a three-dimensional format. For example, the emitter can be configured to emit structured light that can be converted into a topographic surface map. The three-dimensional image of the surgical field can be combined with an overlay of hidden structures (i.e., critical structures) and / or distances to the critical structures. The alert system can provide feedback to the clinician when the critical structure is within a critical distance limit of the surgical device, as further described herein.

[0223] Surgical device 1602 includes an emitter 1606 configured to emit light waves. For example, emitter 1606 can be configured to emit tissue-penetrating infrared wavelengths that are configured to penetrate tissue 1603 and reach critical structure 1601. Emitter 1606 can include, for example, a spectral light source that can be configured to emit, for example, hyperspectral, multispectral, and / or selective spectral waveforms. Emitter 1606 can further emit a structured light pattern that is detected by camera 1620 to generate a topographical surface map of surface 1605.

[0224] The camera 1620 also includes an image sensor or receiver 1608 that is configured to detect light waves emitted from the transmitter 1606 and reflected by the critical structure 1601. The spectral imaging logic component can identify the critical structure based on the reflected light waves received by the image sensor 1608, as further described herein. In various embodiments, the detected critical structure 1601 can be schematically depicted as a line or other symbol on a three-dimensional rendering of the surface 1605 of the visible tissue 1603. For example, the approximate location of the critical structure 1601 can be conveyed to the clinician as a line on a three-dimensional surface map of the surface 1605. In some cases, the image sensor 1608 is also configured to detect visible light and can optionally record a real-time image and / or video feed of the surgical site and convey the image / video to the imaging system and / or its display screen / monitor.

[0225] The surgical visualization system 1600 is configured to determine a device-surface distance d2 from the distal end 1612 of the surgical device 1602 to the tissue surface, a device-vessel distance d3 from the distal end 1612 of the surgical device 1602 to a vessel below the tissue surface, and a surface-vessel distance / depth d4 of a critical structure below the tissue surface, according to one or more distance determination methods and / or systems described herein. For example, the device-surface distance d2 can be determined by surface mapping logic and a resulting surface map generated from a structured light pattern. Additionally or alternatively, the device-surface distance d2 can be determined by a time-of-flight distance sensing system configured to detect a delay between a transmitted wave and a received wave at the surface 1605 of the target tissue 1603. In various embodiments, the device-surface distance d2 can be a distance to a portion of the surface 1605 that overlays an identified critical structure (such as the critical structure 1601). For example, device-surface distance d2 can be the distance from distal end 1612 of surgical device 1602 to the enhancement line closest to surgical device 1602 on a three-dimensional rendering of surface 1605. Device-vessel distance d3 can be determined by NIR time-of-flight sensing of spectral waves received by camera 1620 and / or one or more image sensors 1608. In other cases, device-vessel distance d3 can be determined by triangulation, a three-dimensional camera, and fluoroscopic illumination. In various cases, surface-vessel depth d1 can be calculated and / or triangulated from real-time distances and / or dimensions obtained by spectral imaging, three-dimensional imaging, and / or surface mapping data determined by ultrasound and / or dimensions obtained in preoperative scans. Alternative distance determination systems are also disclosed herein.

[0226] The surgical visualization system 1600 also includes an imaging system, which may include a camera 1620 and a display screen 1650 ( Figures 29 to 30 ). The imaging system can selectively display different information on the screen 1650. For example, the imaging system can include input controls that allow the clinician to select one or more views, dimensions, and / or other information on the display screen. Figure 28 , an exemplary input control, namely, dial 1660, is shown. Dial 1660 allows the clinician to select the device-surface distance d2, the device-vessel distance d3, or the surface-vessel distance d1. For example, the clinician can rotate dial 1660 between different positions to select different distances. In other cases, the input control may include, for example, one or more inputs, buttons, toggles, switches, and / or a touch screen. In some cases, two or more distances may be selected simultaneously and / or display screen 1650 may switch between different distances and / or views after a preset amount of time.

[0227] Now see Figures 29 to 31, the display screen 1650 is configured to display different views 1652a, 1652b, 1652c of the surgical site based on the clinician's input. For example, depending on the orientation of the dial 1660, the display screen 1650 may display different views and / or information. The dial 1660 and its orientation may also be provided on the display screen 1650. Figure 29 In FIG. 1 , the display screen 1650 depicts a first view 1652a in which the surface-vessel distance d1 can be displayed and / or monitored. Figure 29 , dial 1660 is in a first orientation in which surface-vessel distance d1 is selected. In such a case, distance d1 is monitored and relevant information is displayed on screen 1650. For example, distance d1 can be displayed numerically and / or along a first proximity spectrum 1656a that indicates a warning (such as for thermal and / or force-related considerations) when distance d1 decreases below a predefined threshold. For example, a warning can be provided on screen 1650 and / or via additional visual, audible, and / or tactile signals. Additionally, key structures 1601, which are hidden from the naked eye, are schematically depicted via enhanced first view 1652a.

[0228] exist Figure 30 In FIG. 1 , display screen 1650 depicts a second view 1652 b in which device-surface distance d2 may be displayed and / or monitored. Figure 30 , the dial 1660 is in a second orientation in which the device-to-surface distance d2 is selected. In such a case, the distance d2 is monitored and related information is displayed on the screen 1650. For example, the distance d2 can be displayed numerically and / or along a second proximity spectrum 1656b that indicates a warning (such as for thermal and / or force-related considerations) when the distance d2 decreases below a predefined threshold. For example, the warning can be provided on the screen 1650 and / or via additional visual, audible, and / or tactile signals. When the distance to the tip of the surgical device 1602 is selected, the distance can be the distance from the tool's projection point to the tissue surface, which can be determined by a three-dimensional Cartesian coordinate system from the robotic control arm.

[0229] Spectral imaging is used to identify key structures 1601 that are hidden from the naked eye and are schematically depicted via an enhanced second view 1652b. In various cases, the enhanced representation of the key structures 1601 can be switched on and off and / or as shown. Figure 30 is shown as a shaded / dashed background shape. For example, due to Figure 30The selected distance d2 in is not directly to the key structure 1601, so it may be desirable to display the key structure 1601 as a shadow or hidden to avoid distracting the clinician's attention and / or to focus the clinician's attention on other selected information. In various cases, as described herein, the distance d2 can be the distance to the enhanced line on the surface 1605 representing the hidden key structure 1601, and the hidden key structure 1601 can be schematically depicted as a shadow and / or background image for context.

[0230] exist Figure 31 , display screen 1650 depicts a third view 1652c in which the device-vessel distance d3 can be displayed and / or monitored. Figure 31 , dial 1660 is in a third position in which device-vessel distance d3 is selected. In such a case, distance d3 is monitored and relevant information is displayed on screen 1650. For example, distance d3 can be displayed numerically and / or along a third proximity spectrum 1656c that indicates a warning (such as for thermal and / or force-related considerations) when distance d3 decreases below a predefined threshold. For example, a warning can be provided on screen 1650 and / or via additional visual, audible, and / or tactile signals.

[0231] The enhanced third view 1652c schematically depicts the key structures 1601 hidden from the naked eye. In various cases, the enhanced representation of the surface 1605 of the tissue 1603 can be switched on and off and / or as shown in FIG. Figure 31 1605 is shown as a shaded / dashed shape. For example, because the selected distance d3 is independent of the orientation of the surface 1605, it may be desirable to display the surface 1605 as shaded or hidden to avoid distracting the clinician and / or to focus the clinician's attention on other information. In some cases, a three-dimensional surface map can be generated by structured light and surface mapping logic, which can be used to selectively display the surface 1605 of the tissue 1603 on the display screen 1650, as further described herein.

[0232] Figures 29 to 31 The views and / or portions thereof can be toggled on or off. In various instances, the surgical visualization system 1600 can operate without adding contrast agents to any structures. Selective spectral imaging can allow for selectivity of specific targets, which can be paired with specific distance and proximity alerts based on identification of critical structures (e.g., such as nerves, organs, ureters, veins, arteries, or lymph nodes) and, for example, the boundaries or mechanical margins of a tumor.

[0233] When manipulating a surgical device around a surgical site, a clinician may want to know the position of the surgical device and / or "see" the position of the surgical device relative to one or more other structures (e.g., surface tissue, hidden critical structures, other surgical devices, etc.). However, the surgical device or a portion thereof may be obscured and invisible during surgery. For example, tissue or another anatomical structure may be positioned between the clinician's observation point (e.g., a camera) and the surgical device or a portion of the surgical device (such as an articulation joint and / or jaws), which may block the clinician's field of view. In such cases, the clinician may not be able to visualize the surgical device or a portion thereof and may risk moving the surgical device too close to a critical structure. For example, the clinician may accidentally manipulate a surgical stapler, dissector, energy device, and / or needle too close to a critical structure and risk dissecting or otherwise injuring the critical structure.

[0234] In various cases, the surgical visualization system can allow the clinician to visualize occluded / partially occluded surgical devices relative to critical structures. For example, the surgical visualization system can be configured to identify surgical devices. In various cases, spectral imaging (such as hyperspectral, multispectral, or selective spectral imaging) can be utilized to identify surgical devices. Other detection modalities include:

[0235] For example, ultrasound, registered magnetic resonance imaging (MRI), and computed tomography (CT) scans.The surgical visualization system may further include an imaging system including a display operably configured to depict the identified surgical device relative to anatomy and / or other surgical devices.

[0236] In such cases, the clinician can track the position of the surgical device relative to the critical structure even when the surgical device and / or the critical structure are obscured (or partially obscured) from view. Visualization of the hidden surgical device relative to the critical structure can allow the clinician to carefully and quickly maneuver the surgical device within a desired proximity to the critical structure. For example, the clinician can ensure that the dissector remains a sufficient distance from the critical structure (e.g., from an artery). In another example, the clinician can ensure that the biopsy needle reaches one or more appropriate locations within the critical structure (e.g., within a tumor).

[0237] Now see Figure 32 and Figure 33 , showing a surgical visualization system 1800. The surgical visualization system 1800 can be similar in many respects to the surgical visualization system 100 ( Figure 1). For example, the surgical visualization system 1800 can be configured to identify one or more critical structures embedded in tissue or otherwise hidden from view, and determine one or more distances relative to the visible tissue and / or critical structures. The surgical visualization system 1800 includes a hyperspectral camera 1820 that includes an image sensor, as further described herein. For example, the hyperspectral camera 1820 includes an emitter 1806 and a receiver 1808. The emitter 1806 is configured to emit multiple tissue-penetrating wavelengths. In various instances, the emitter 1806 can be configured to emit multiple hyperspectral waveforms, multispectral waveforms, or selective spectral waveforms that are configured to penetrate tissue and reach one or more critical structures, such as a surgical device or another anatomical structure. For example, the emitter 1806 is configured to emit waveforms that penetrate tissue 1803a and 1803b.

[0238] The present invention further describes the identification of hidden anatomical structures (such as nerves, blood vessels, or ureters). Furthermore, in addition to identifying hidden anatomical structures, spectral imaging can be configured to detect metal, such as metal portions of end effectors, metal shafts, staples, and / or metal bands or plates. In such cases, the surgical visualization system can further detect hidden surgical devices. For example, receiver 1808 on hyperspectral camera 1820 is configured to identify critical structures including surgical devices, such as first device 1802a positioned through first trocar 1810a and second device 1802b positioned through second trocar 1810b. First device 1802a is a robotic grasper tool including end effector 1812a, which is completely hidden from the clinician's view by tissue 1803a, in this example, a portion of the colon beneath which end effector 1812a is positioned. Second device 1802b is a robotic suturing tool including end effector 1812b, which is partially hidden from the clinician's view by tissue 1803a. For example, the end effector 1812b is positioned to clamp tissue 1803a, and thus the first jaw is positioned above the tissue 1803a and the second jaw is positioned below the tissue 1803a.

[0239] In some cases, receiver 1808 can identify the shaft of devices 1802a, 1802b and / or the end effectors 1812a, 1812b of devices 1802a, 1802b, respectively. In one aspect, a clinician can select or mark the shaft or end effectors 1812a, 1812b, or another portion of devices 1802a, 1802b, to be tracked during a surgical procedure. For example, a clinician can select to track the shaft of one of devices 1802a, 1802b to obtain positional information about the side of the tool relative to other adjacent anatomical structures. Additionally or alternatively, receiver 1808 can identify one or more anatomical structures, such as artery 1801 embedded in tissue 1803b. Marking of structures is further described in concurrently filed U.S. patent application Ser. No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MOITORING,” which is hereby incorporated by reference in its entirety.

[0240] The receiver 1808 is configured to be able to track the position of key structures (e.g., end effectors 1812a, 1812b and artery 1801) intraoperatively. In various embodiments, the surgical visualization system 1800 is configured for use during minimally invasive surgical procedures such as Figure 32 The surgical visualization system 1800 also includes an imaging system including a camera 1820 and a display 1850 ( Figure 33 ). The location of critical structures can be conveyed to the clinician via the imaging system's display 1850.

[0241] Now see Figure 33 Display 1850 is a monitor that is configured to display a real-time video feed of the surgical site. For example, images from camera 1820 can be transmitted to display 1850 during surgery to provide the clinician with a real-time view of the surgical site. In addition, the view of the surgical site can be enhanced with additional information, including hidden critical structures and / or distances. Display 1850 shows an embedded artery 1801 and surgical devices 1802a, 1802b, including their hidden portions enhanced into the view of the surgical site.

[0242] In various cases, the display 1850 may also depict a three-dimensional rendering of certain anatomical structures at the surgical site. For example, tissue 1803a (a portion of the colon) may be depicted on the display 1850. The rendering of the colon may be obtained via structured light and surface mapping logic, as further described herein. For example, the hyperspectral camera 1820 may also include a structured light source that can, for example, emit a structured light pattern onto the tissue 1803a and / or the tissue 1803b. In addition, the hyperspectral camera 1820 may also include a receiver configured to detect the structured light pattern. See Figure 33 In the view in FIG, the colon is shown as a shadow / background image obtained from the structured light and surface mapping logic components, and the colon on the display 1850 is superimposed with key structures (e.g., end effectors 1812a, 1812b and artery 1801) identified by the spectral imaging system (e.g., hyperspectral camera 1820 and spectral imaging recognition logic components).

[0243] Display 1850 also includes a proximity spectral indicator 1856 that communicates the proximity of one or both surgical devices 1802a, 1802b to artery 1801. In some cases, proximity spectral indicator 1856 may display the proximity of the closest surgical device 1802a, 1802b. In other cases, the proximity of the closest surgical device 1802a, 1802b may be the default mode; however, the clinician may selectively select another surgical device and / or anatomical structure. In other cases, proximity spectral indicator 1856 may alternate and / or switch between different surgical devices 1802a, 1802b, or may include different spectral indicators 1856 for different surgical devices 1802a, 1802b.

[0244] The proximity spectrum indicator 1856 can utilize color (e.g., red, yellow, green), size, and / or other symbols to convey the proximity of the surgical devices 1802a, 1802b to the artery 1801. For example, a proximity zone can be defined around the artery 1801 and assigned a color or a series of colors. As the distance changes during surgery, the marker 1854 can move along the proximity spectrum indicator 1856 to convey the proximity zone in real time. Figure 33 For example, as the proximity of the nearest surgical device and artery 1801 decreases, marker 1854 may move from a green region to a yellow region to a red region.

[0245] In various instances, the surgical visualization system 1800 is configured to provide a warning, alert, or other indication to the clinician when one of the surgical devices 1802a, 1802b approaches a predefined position range / minimum distance limit / proximity zone around a critical structure, such as the artery 1801. The alert may be provided at different critical distances based on the type of surgical device 1802a, 1802b and the type of critical structure. In one instance, the surgical visualization system 1800 may provide a warning when the marker 1854 moves into a red zone.

[0246] The surgical visualization system 1800 is configured to determine one or more distances d from the end effectors 1812a and 1812b, respectively. a and d bFor example, the surgical visualization system 1800 can utilize time-of-flight distance measurements to determine various distances from the camera 1820 (e.g., utilizing target tissue penetration wavelengths to the surgical devices 1802a, 1802b and the artery 1801, as further described herein). A triangulation algorithm can then determine relative distances between structures, such as the distance d a and d b .

[0247] Surgical visualization systems can be used during biopsy. Ultrasound-assisted biopsy procedures are Figure 38 FIG2 shows an ultrasound device 1920 configured to emit ultrasound waves 1924 directed toward a surgical site. In this example, ultrasound waves 1924 are directed toward a thyroid gland 1903 to identify the location of a nodule 1901 or tumor within the thyroid gland 1903 so that a clinician can guide a biopsy needle 1902 toward the nodule 1901. In various situations, it may be desirable to collect multiple samples within the nodule 1901 and to biopsy tissue from different locations within the nodule 1901. Ultrasound-assisted biopsy is further described in the article "THYROID FINE NEEDLE ASPIRATION (FNA) BIOPSY," which can be found at www.fairview.org / patient-education / 90246.

[0248] In one aspect, a surgical visualization system including spectral imaging technology can be utilized during a biopsy procedure. Figures 34 to 37 , showing a surgical visualization system 2000. The surgical visualization system 2000 can be similar in many respects to the surgical visualization system 100 ( Figure 1 For example, the surgical visualization system 2000 can be configured to identify one or more critical structures embedded in tissue or otherwise hidden from view, and to determine one or more distances relative to visible tissue and / or the critical structures.

[0249] Surgical visualization system 2000 includes a hyperspectral camera 2020, which includes an image sensor, as further described herein. For example, camera 2020 includes a transmitter 2006 and a receiver 2008. Transmitter 2006 is configured to emit multiple tissue-penetrating waves. In various embodiments, transmitter 2006 can be configured to emit multiple hyperspectral waveforms, multispectral waveforms, or selective spectral waveforms configured to penetrate tissue and reach one or more critical structures, such as needle 2002 and tumor 2001. For example, transmitter 2006 is configured to emit a waveform that penetrates the surface 2005 of tissue 2003, such as the thyroid gland. Given the different molecular composition of needle 2002 (which is typically composed of metal and / or plastic) and tissue, spectral imaging can distinguish an embedded needle 2002 from tissue 2003 and tumor 2001, as the spectral signatures of these materials will be different. Furthermore, as further described herein, spectral imaging can distinguish between different types of tissue (i.e., thyroid gland 2003 and tumor 2001 embedded therein). The surgical visualization system 2000 is configured to communicate and / or determine the proximity of the aspiration needle 2002 to tissue (e.g., tissue of the thyroid gland 2003) and / or a critical structure (e.g., tumor 2001). For example, the surgical visualization system 2000 can use the spectral signature differences between the aspiration needle 2002, the tissue 2003, and the target tumor 2001 to determine the depth of the aspiration needle 2002 relative to the tissue 2003 and the target critical structure (i.e., tumor 2001). Therefore, the surgical visualization system 2000 can be used instead of the ultrasound device 1920 ( Figure 38 ).

[0250] The surgical visualization system 2000 also includes a structured light source and surface mapping logic, as further described herein. In one aspect, the camera 2020 includes a structured light source. For example, the emitter 2006 can be configured to selectively pulse between a spectral imaging waveform and a structured light pattern. In certain aspects, the camera 2020 is further configured to detect a pattern of structured light on the surface 2005 of the tissue 2003. The surface mapping logic is configured to generate a three-dimensional model or rendering of the tissue surface 2005, which can be provided to the imaging system and overlaid with data from the spectral imaging system, as further described herein.

[0251] exist Figure 34 In FIG, the tip 2012 of the needle 2002 is located at a distance d1 from the surface 2005 of the anatomical target, i.e., tissue 2003, and at a distance d2 from the surface of the embedded tumor 2001. Figure 36In FIG. 2 , the tip 2012 of the needle 2002 is advanced into the tumor 2001 to a distance d2 from the surface of the tumor 2001. The position of the needle 2002 relative to the tissue 2003 and the tumor 2001 is transmitted to the imaging system. For example, the surgical visualization system 2000 may include an imaging system including a camera 2020 and a display 2050 ( Figure 35 and Figure 37 ). Camera 2020 includes an image sensor, as further described herein. The location of the identified structures (e.g., tumor 2001 and needle 2002) can be communicated to the clinician via display 2050.

[0252] Display 2050 is a monitor configured to display a real-time video feed of the surgical site. For example, images from camera 2020 can be transmitted to display 2050 during surgery to provide the clinician with a real-time view of the surgical site. The view of the surgical site can be enhanced with additional information, including hidden critical structures and / or distances. For example, display 2050 shows an embedded tumor 2001 and needle 2002, including the portion of needle 2002 that is hidden by tissue and not visible to the clinician.

[0253] In various cases, the display 2050 also depicts a three-dimensional rendering of certain anatomical structures at the surgical site. For example, tissue 2003 is shown on the display 2050. The rendering of the anatomical target tissue 2003 can be obtained via structured light and surface mapping logic components, as further described herein. For example, the hyperspectral camera 2020 can also include a structured light source that can emit a structured light pattern onto the tissue 2003. In addition, the hyperspectral camera 2020 can include a receiver configured to detect the structured light pattern. See Figure 35 and Figure 37 In the view on display 2050 , tissue 2003 and embedded key structures (eg, needle 2002 and tumor 2001 ) are overlaid.

[0254] The display 2050 also includes proximity spectrum indicators 2056a, 2056b that indicate the proximity of the delivery needle 2002 to the surface 2005 of the tissue 2003 (distance d1) and to the surface of the embedded tumor 2001 (distance d2). For example, the proximity spectrum indicators 2056a, 2056b may define the distance from the surface as positive and negative values. Figure 34 In FIG. 2 , the needle 2002 is inside the tissue 2003 and outside the tumor 2001. Therefore, in FIG. Figure 35 In FIG, the proximity spectrum indicator 2056a indicates that the distance d1 is a negative value, and the proximity spectrum indicator 2056b indicates that the distance d2 is a positive value. Figure 36In FIG. 2 , needle 2002 is inside tissue 2003 and inside tumor 2001, so in FIG. Figure 37 , the proximity spectrum indicator 2056a indicates that the distance d1 is a negative value, and the proximity spectrum indicator 2056b indicates that the distance d2 is also a negative value.

[0255] In various aspects, the proximity spectrum indicators 2056a, 2056b may include colors, numerical values ​​or ranges and / or other symbols to detect proximity. In some cases, a single proximity spectrum indicator may alternate and / or switch between different distances d1 and d2, and / or the clinician may select one or more distances to be displayed.

[0256] In various circumstances, the surgical visualization system 2000 is configured to provide an alert, warning, or other indication to the clinician when the needle 2002 approaches a predefined range of positions, minimum distance limits, and / or proximity zones around the tumor 2001 and / or depth within the tissue 2003 .

[0257] The surgical visualization system 2000 is configured to determine one or more distances using a time-of-flight sensor system (e.g., using a target wavelength to the surface 2005, the needle 2002, and the tumor 2001, as further described herein). Additionally, a triangulation algorithm can determine relative distances between structures, such as distances d1 and d2.

[0258] In some cases, Figures 34 to 37 The biopsy procedure shown can be enhanced with contact-based ultrasound guidance, e.g. Figure 38 shown.

[0259] Various surgical visualization systems disclosed herein are configured to be able to identify one or more key structures that are embedded in the tissue or otherwise hide invisible, and determine one or more distances relative to visible tissue and / or key structures. In some cases, the key structure can be a surgical device, such as a grasper, a dissector, a stapler, a nail, a nail row / line, a circular stapler, a circular stapler anvil, a gastric bougie or a hernia tack / clamp. During surgery, it may be desirable to track the orientation of these key structures. For example, subsequent steps in surgery can depend on the orientation of the key structures that were implanted and / or moved during an earlier step.

[0260] As an example, during a low anterior resection (LAR) of the colon, the clinician may want to identify and track staple lines along the ends of the colon before achieving anastomosis of the ends. Figures 39 to 41In an exemplary LAR procedure, a colon 2270 can be transected into two parts, a first colon part 2272 and a second colon part 2274, to remove an intermediate portion, such as a cancerous tumor. A staple line can seal the terminal ends of the first colon part 2272 and / or the second colon part 2274 before resection.

[0261] The first colon portion 2272 and the second colon portion 2274 can be removed using a circular stapler 2202. Figure 39 As shown, the circular stapler 2202 is positioned in the first colon portion 2272 and the anvil 2204 is positioned in the second colon portion 2274. For example, the circular stapler 2202 can be inserted through the anus and the anvil 2204 can be positioned through the incision. The anvil 2204 and the circular stapler 2202 are aligned, and the anvil 2204 can be fixed to the circular stapler 2202, as shown in FIG. Figure 40 For example, the trocar 2206 of the circular stapler 2202 ( Figure 39 ) can be positioned on the shaft 2208 of the anvil 2204 ( Figure 39 ). Then, see Figure 41 , the circular stapler 2202 fires the blade and staples against the anvil 2204 to form a sealed path between the first colon portion 2272 and the second colon portion 2274. The LAR procedure is further described in "OPEN TECHNIQUE FOR LOW ANTERIOR RESECTION," which can be found at abdominalkey.com / open-technique-for-low-anterior-resection.

[0262] Circular stapler 2202 and the suitable alignment of anvil 2204 are important for sealing the first colon portion 2272 and the second colon portion 2274. Therefore, visualizing circular stapler 2202 and / or anvil 2204 during surgery may be helpful. Visualizing the orientation of staple line and determining one or more distances relative to staple line may also be helpful. For example, the clinician may want to track the proximity of circular stapler 2202, anvil 2204 and one or more staple lines (and therefore the cut-off end of colon 2270) during surgery to facilitate the alignment and / or positioning of device. For example, surgical visualization system as disclosed herein may be utilized to image, visualize and track the orientation of circular stapler 2202, trocar 2206 and anvil 2204 and trocar 2206 relative to the orientation of the desired exit point of colon 2270.

[0263] Now see Figures 42 to 45 , depicting a surgical visualization system 2300. The surgical visualization system 2300 can be similar in many respects to the surgical visualization system 100 ( Figure 1 ). For example, the surgical visualization system 2300 can be configured to identify one or more critical structures embedded in tissue or otherwise hidden from view, and determine one or more distances relative to the visible tissue and / or critical structures. The surgical visualization system 2300 includes a hyperspectral camera 2320, which includes an image sensor, as further described herein. For example, the camera 2320 includes an emitter 2306 and a receiver 2308. The emitter 2306 is configured to emit multiple tissue penetrating waves. In various embodiments, the emitter 2306 can be configured to emit multiple hyperspectral waveforms, multispectral waveforms, or selective spectral waveforms that are configured to penetrate tissue and reach one or more critical structures, such as a surgical device or another anatomical structure. For example, the emitter 2306 is configured to emit a waveform that penetrates the colon 2303. The surgical visualization system 2300 also includes a structured light source.

[0264] The present invention further describes the identification of hidden anatomical structures (such as nerves, blood vessels, or ureters) and surgical devices (such as surgical end effectors, shafts, or staples). For example, the receiver 2308 on the hyperspectral camera 2320 is configured to be able to identify key structures including surgical devices, such as the first device 2302a and the second device 2302b. The first device 2302a is a robotic grasper tool, and the second device 2302b is a robotic suturing tool. In other cases, the devices 2302a and 2302b may be handheld surgical instruments and may be used, for example, in laparoscopic surgery. In some cases, the receiver 2308 may identify the shafts and / or end effectors of the devices 2302a and 2302b. In addition, the receiver 2308 on the hyperspectral camera 2320 is configured to be able to identify staple lines 2380a and 2380b in the colon 2303. In one aspect, a clinician can select or mark a portion of a critical structure (e.g., surgical device 2302a, 2302b and / or staple lines 2380a, 2380b) for tracking during surgery. Marking of structures is further described in concurrently filed U.S. patent application Ser. No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MONITORING,” which is hereby incorporated by reference in its entirety.

[0265] The receiver 2308 is configured to be able to track the position of key structures (e.g., surgical devices 2302a, 2302b and staple lines 2380a, 2380b) intraoperatively. In various cases, the surgical visualization system 2300 is configured to be able to be used during LAR surgery, such as Figures 39 to 41In such procedures, it may be helpful to visualize the obscured or partially obscured surgical devices 2302a, 2302b relative to the staple lines 2380a, 2380b, which may also be obscured or partially obscured. For example, one or more critical structures may be obscured by fat or mesentery. The surgical visualization system 2300 also includes an imaging system that includes a camera 2320 and a display 2350 ( Figure 43 and Figure 45 The relative positions of the surgical devices 2302a, 2302b and the staple lines 2380a, 2380b can be conveyed to the clinician via the display 2350 of the imaging system.

[0266] Now see Figure 43 , display 2350 is a monitor that is configured to display a video feed of the surgical site in real time. For example, images from camera 2320 can be transmitted to display 2350 during surgery to provide the clinician with a real-time view of the surgical site. In addition, the view of the surgical site can be enhanced with additional information including hidden critical structures and / or distances. Display 2350 shows obscured staple lines 2380a, 2380b and surgical devices 2302a, 2302b, including their hidden portions enhanced in the view of the surgical site.

[0267] In various cases, the display 2350 also depicts a three-dimensional rendering of certain anatomical structures at the surgical site. For example, the colon 2303 can be depicted on the display 2350. The rendering of the colon 2303 and its movement can be obtained via structured light and surface mapping logic, as further described herein. For example, the hyperspectral camera 2320 can also include a structured light source that can, for example, emit a structured light pattern onto the colon 2303. In addition, the hyperspectral camera 2320 can also include a receiver configured to detect the structured light pattern. In other cases, a separate device can emit and / or detect the structured light pattern. See Figure 43 and Figure 45 In the view in FIG, the colon 2303 is shown as a shadow / background image obtained from the structured light and surface mapping logic components, and a three-dimensional rendering of the colon 2303 on the display 2350 is superimposed with key structures identified by spectral imaging (e.g., surgical devices 2302a, 2302b and staple lines 2380a, 2380b).

[0268] The display 2350 also includes a proximity spectrum indicator 2356 that communicates the proximity of one or both of the surgical devices 2302a, 2302b to one or both of the staple lines 2380a, 2380b. In various circumstances, the clinician can selectively select staple lines 2380a, 2380b and surgical devices 2302a, 2302b to be tracked and monitored on the proximity spectrum indicator 2356 and / or displayed on the display 2350. In other circumstances, the proximity spectrum indicator 2356 can alternate and / or switch between different surgical devices 2302a, 2302b and / or staple lines 2380a, 2380b, or can include different spectral indicators 2356 for different surgical devices 2302a, 2302b and staple lines 2380a, 2380b. In some cases, the marked staple lines 2380a, 2380b may be tracked by the camera 2320 and remain visible on the display 2350, for example.

[0269] The proximity spectrum indicator 2356 can utilize color (e.g., red, yellow, green), size, and / or other symbols to convey the proximity of the surgical device 2302a, 2302b to the staple lines 2380a, 2380b. For example, a proximity zone can be defined around the staple lines 2380a, 2380b, and the proximity zone can be assigned a color or a series of colors. As the distance changes during surgery, the marker 2354 can move along the proximity spectrum indicator 2356 to convey the proximity zone in real time. Still referring to Figure 43 and Figure 45 For example, as the proximity decreases, the marker 2354 may move from the green area to the yellow area to the red area.

[0270] In various embodiments, the surgical visualization system 2300 is configured to provide a warning, alert, or other indication to the clinician when one of the surgical devices 2302a, 2302b approaches a predefined range of positions, minimum distance limits, and / or proximity zones around the staple lines 2380a, 2380b. The alert may be provided at different critical distances based on the type of surgical device 2302a, 2302b. In one embodiment, the surgical visualization system 2300 may provide a warning when the marker 2354 moves into a red zone.

[0271] The surgical visualization system 2300 is configured to determine one or more distances from each of the surgical devices 2302a and 2302b. For example, the surgical visualization system 2300 can utilize time-of-flight distance measurements to determine various distances from the camera 2320 (e.g., utilizing target wavelengths to the surgical devices 2302a, 2302b and staple lines 2380a, 2380b, as further described herein). For example, a triangulation algorithm can then determine the relative distances between the structures.

[0272] In some cases, the robotic system can be configured to autonomously track the marked staple lines 2380a, 2380b, as if grasping the staple lines and positioning the colon for subsequent surgical steps (such as anastomosis). For example, the robotic system can automatically move to a position at a preset distance away from and / or above the staple lines 2380a, 2380b. While automatically moving near the staple lines 2380a, 2380b, the clinician at the robotic console can also position the surgical tools via controls at the command station / console.

[0273] For example, the anastomosis of colon 2303 is Figure 44 and Figure 45 Shown in. Figure 44 2 shows a surgical device 2302 including a circular stapler 2302c and anvil 2302d for anastomosis of a colon 2303. For example, the robotic system may control the surgical device 2302a to grab the staple line 2380a and pull the staple line 2380a, wherein the anvil 2302d of the surgical device 2302 is directed toward the circular stapler 2302c. In such a case, the hyperspectral camera 2320 is configured to be able to detect the circular stapler 2302c hidden in the lower portion of the colon 2303 and the anvil 2302d hidden in the upper portion of the colon 2303. In some cases, visualization of the anvil 2302d and the circular stapler 2302c may facilitate alignment of the anvil 2302d with the trocar 2312 of the circular stapler 2302c.

[0274] Now see Figure 45 , shows a display 2350 of the surgical visualization system 2300. The display 2350 shows the obscured staple lines 2380a, 2380b, the surgical device 2302a, the circular stapler 2302c, the anvil 2302d, and another grasper device 2302e (including hidden portions thereof) superimposed on a surface map of visible tissue of the colon 2303. The display 2350 is further enhanced with a distance d1 from the anvil 2302d to the first staple line 2380a and a distance d2 from the anvil 2302d to the trocar 2312 of the circular stapler 2302c.

[0275] When viewed from the laparoscope, the surgical device 2302 (i.e., circular stapler 2302c and anvil 2302d) within the colon 2303 can be viewed using the surgical visualization system 2300. In various situations, the clinician can switch on or off the visibility of the surgical device 2302 or other surgical devices. When the surgical device 2302 is visualized, the clinician can see the position of the circular stapler 2302c, the trocar 2312, and the anvil 2302d. In addition, the clinician can visualize the position of the trocar 2312 relative to the desired exit point of the colon 2303.

[0276] Now see Figure 46 , shows a stomach 2403 with a surgical device positioned therein, more specifically, a bougie 2402. Bougie 2402 is a surgical device that is generally constructed of a flexible or conformable body 2412. Bougie 2402 may also include one or more bands 2414a, 2414b, 2414c, 2414d, 2414e, 2414f, 2414g, which may be constructed of rigid plastic or metal. During a sleeve gastrectomy, bougie 2402 may be positioned in stomach 2403 along the lesser curvature of stomach 2403 and may remove an adjacent portion of the stomach, typically the fundus 2405. For example, staple lines 2416 may dissect stomach 2403 and remove fundus 2405 from the remaining portion of stomach 2403.

[0277] Now see Figure 47, shows the suturing step of a sleeve gastrectomy. During the suturing step, a linear stapler 2502a and a grasper 2502b impact gastric tissue 2503. Stapler 2502a and grasper 2502b may, for example, be robotic tools during robotic surgery. Critical structure 2501 (such as the pylorus) is at least partially hidden by tissue 2503. In addition, a bougie 2512 is positioned within gastric tissue 2503. In various circumstances, the surgeon may want to know the orientation of linear stapler 2502a, grasper 2502b, and / or bougie 2512 relative to critical structure 2501. For example, it may be important to separate bougie 2512 from critical structure 2501 by a minimum distance α, such as at least 5 cm from the pylorus. In such circumstances, critical structure 2501 constitutes an anatomical landmark for positioning bougie 2512. In various cases, detectable key structures (such as key structure 2501 and / or bougie 2512) can serve as anatomical landmarks for positioning other surgical devices. Other distances can also be monitored and / or visualized to ensure the proper positioning of bougie 2512 relative to various anatomical structures. For example, it may be important to position linear stapler 2502a at an appropriate distance from bougie 2512 for properly sizing the gastric sleeve while preventing excessive strain on the staples. Gastric sleeve surgery is further described in the following article: "Sleeve Gastrectomy Surgical Assistive Instrument for Accurate Remnant Stomach Volume", ASME, J. Med. Devices, 2010; Vol. 4, No. 2, which is incorporated herein by reference in its entirety at medicaldevices.asmedigitalcollection.asme.org / article.aspx?articleid=1474028. Gastric sleeve surgery is further described at aischannel.com / society / main-steps-to-perform-a-sleeve-gastrectomy / .

[0278] Now see Figure 48 and Figure 49 , depicting a surgical visualization system 2600. The surgical visualization system 2600 can be similar in many respects to the surgical visualization system 100 ( Figure 1). For example, the surgical visualization system 2600 can be configured to identify one or more critical structures embedded in tissue or otherwise hidden from view and determine one or more distances relative to visible tissue and / or critical structures. Specifically, the surgical visualization system 2600 can be configured to allow visualization of a hidden bougie 2612 positioned within the stomach 2603 during a gastric sleeve procedure. The bougie 2612 includes a flexible inflatable body or sleeve 2614 and a band 2616. In other cases, the bougie 2612 can have more or less than Figure 48 Five bands 2616 are shown. Bands 2616 are constructed of a different material than body 2614. As further described herein, spectral imaging can be configured to identify the different materials and, therefore, the location of bougie 2612 even when bougie 2612 is concealed within stomach 2603. For example, bougie 2612 may be constructed of optically different materials at key locations, which can be detected using spectral imaging.

[0279] Surgical visualization system 2600 includes a hyperspectral camera 2620, which includes an image sensor, as further described herein. For example, camera 2620 may include a transmitter 2606 and a receiver 2608. Camera 2620 may be utilized during laparoscopic surgery to image a surgical site. Transmitter 2606 is configured to emit multiple tissue-penetrating waves. In various embodiments, transmitter 2606 may be configured to emit multiple hyperspectral waveforms, multispectral waveforms, or selective spectral waveforms that are configured to penetrate tissue and reach one or more critical structures, such as surgical devices or another anatomical structure. For example, transmitter 2606 may be configured to emit waveforms that penetrate stomach 2603. Surgical visualization system 2600 may also include a structured light source and a receiver / image sensor 2068 that may be configured to determine the topography of stomach 2603.

[0280] The present invention further describes the identification of hidden anatomical structures (such as nerves, blood vessels, or ureters) and surgical devices (such as surgical end effectors, shafts, or staples). For example, the receiver 2608 on the hyperspectral camera 2620 is configured to be able to identify and track key structures, including surgical devices such as the bougie 2612, the body 2614, and the band 2616, as well as surgical tools such as the linear stapler 2602.

[0281] The surgical visualization system 2600 also includes an imaging system including a camera 2620 and a display 2650 ( Figure 49 The relative positions of the linear stapler 2602 and the bougie 2612 can be communicated to the clinician via the display 2650 of the imaging system.

[0282] Now see Figure 49, display 2650 is a monitor that is configured to display a video feed of the surgical site in real time. For example, images from camera 2620 can be transmitted to display 2650 during surgery to provide the clinician with a real-time view of the surgical site. In addition, the view of the surgical site can be enhanced with additional information including hidden key structures and / or distances. Display 2650 shows an obscured bougie 2612 enhanced into the view of the surgical site. In various cases, the visualization of obscured key structures (such as bougie 2612) can be toggled on and off by the clinician. For example, bougie 2612 can be depicted as a shaded image in the default view and can be selectively removed from display 2650 by specific user input.

[0283] In various cases, the display 2650 also depicts a three-dimensional rendering of certain anatomical structures at the surgical site. For example, a stomach 2603 may be depicted on the display 2650. The rendering of the stomach 2603 may be obtained via structured light and surface mapping logic, as further described herein. Figure 49 , the stomach 2603 is shown as a shadow / background image obtained from the structured light and surface mapping logic components, and key structures (e.g., bougie 2612 and linear stitcher 2602) are superimposed on the background. In other words, the spectral image of the hidden structure is integrated with the three-dimensional representation of the stomach 2603.

[0284] The surgical visualization system 2600 is configured to determine one or more distances from the linear stapler 2602 and / or the bougie 2612 to a key structure (such as the pylorus, i.e., the opening from the stomach 2603 into the small intestine). The surgical visualization system 2600 can utilize time-of-flight distance measurements to determine various distances from the camera 2620 (e.g., utilizing a target wavelength to the surgical device and / or anatomical structure, as further described herein). For example, a triangulation algorithm can then determine the relative distances between the structures. In various cases, an alert can be provided when the distance meets a threshold and / or range, as further described herein.

[0285] In various cases, different materials of bougie 2612 can assist in the positioning of linear stapler 2602. For example, in certain gastric sleeve surgeries, a haptic can provide feedback to the clinician regarding the placement of linear stapler 2602. For example, the clinician can palpate stomach 2603 to determine the orientation of bougie 2612. However, in robotic applications, the clinician may not be able to fully assess the orientation of bougie 6212 based on the haptic / tactile feedback. The visualization of bougie 2612 can provide a replacement and / or supplement for the haptic. For example, surgical visualization system 2600 is configured to be able to determine the distance from the distal end of linear stapler 2602 to bougie 2612, which can help the robotic system locate various medical devices. In addition, the distance and / or orientation can be communicated to the clinician by an imaging system, as further described herein.

[0286] In various aspects, a surgical visualization system can be configured to identify, for example, metal fasteners, such as clamps. For example, a surgical clamp can be embedded in tissue during a surgical procedure (e.g., placed over a vein or artery to occlude the tissue). Firing a staple against the surgical clamp can affect the resulting firing motion. For example, a surgical clamp between the jaws of an end effector can prevent the end effector from evenly clamping the tissue and / or can jam the closure beam, firing member, and / or cutting element. Additionally, if a staple is fired against the surgical clamp, the staple may be misfired and / or deformed. In such instances, the staple line may result in an imperfect seal. Various surgical visualization systems disclosed herein can identify surgical clamps and, in various instances, notify a clinician when a surgical clamp is positioned too close to a surgical device (e.g., in or near a transection position).

[0287] Now see Figure 50 and Figure 51 , showing a surgical visualization system 2700. The surgical visualization system 2700 can be similar in many respects to the surgical visualization system 100 ( Figure 1 ). For example, the surgical visualization system 2700 can be configured to identify one or more critical structures embedded in tissue or otherwise hidden from view, and to determine one or more distances relative to visible tissue and / or critical structures. Specifically, the surgical visualization system 2700 is configured to allow visualization of a hidden clamp 2712 within tissue 2703 during a surgical procedure. As further described herein, spectral imaging can be configured to identify the material of the clamp 2712, and therefore the orientation of the clamp 2712, even when the clamp 2712 is hidden from view.

[0288] The surgical visualization system 2700 includes a hyperspectral camera 2720 that includes an image sensor, as further described herein. For example, the camera 2720 includes a transmitter 2706 and a receiver 2708. The camera 2720 can be used during laparoscopic surgery to image a surgical site. The transmitter 2706 is configured to emit multiple tissue-penetrating waves. In various embodiments, the transmitter 2706 can be configured to emit multiple hyperspectral waveforms, multispectral waveforms, or selective spectral waveforms that are configured to penetrate tissue and reach one or more critical structures, such as a surgical device or another anatomical structure. For example, the transmitter 2706 is configured to emit a waveform that penetrates tissue 2703. The surgical visualization system 2700 also includes a structured light source that can be configured to determine the surface topography of the tissue 2703.

[0289] The present invention further describes the identification of hidden anatomical structures (such as nerves, blood vessels, or ureters) and surgical devices (such as surgical end effectors, shafts, or nails). For example, the receiver 2708 on the hyperspectral camera 2720 is configured to be able to identify and track key structures, including anatomical structures (such as blood vessels 2701), surgical devices (such as clamps 2712), and surgical tools (such as linear staplers 2702a and graspers 2702b).

[0290] The surgical visualization system 2700 also includes an imaging system comprising a camera 2720 and a display 2750 ( Figure 51 The relative positions of the surgical devices 2702a, 2702b, the clamp 2712, and the blood vessel 2701 may be communicated to the clinician via the display 2750 of the imaging system.

[0291] Now see Figure 51 , display 2750 is a monitor that is configured to display a video feed of the surgical site in real time. For example, images from camera 2720 can be transmitted to display 2750 during surgery to provide the clinician with a real-time view of the surgical site. In addition, the view of the surgical site can be enhanced with additional information including hidden key structures and / or distances. Display 2750 shows an obscured fixture 2712 enhanced to the view of the surgical site. In various cases, the visualization of obscured key structures (such as fixture 2712) can be toggled on and off by the clinician. For example, fixture 2712 can be depicted as a shadow in the default view and can be selectively removed from the view on display 2750 by specific user input.

[0292] In various cases, the display 2750 also depicts a three-dimensional rendering of certain anatomical structures at the surgical site. For example, tissue 2703 may be depicted on the display 2750. The rendering of the tissue 2703 may be obtained via structured light and surface mapping logic, as further described herein. Figure 51 In the view on display 2750 in FIG, tissue 2703 is shown as a shadow / background image obtained from the structured light and surface mapping logic components, with key structures identified using spectral imaging (e.g., clamp 2712, blood vessel 2701, and surgical devices 2702a, 2702b) superimposed on the background. In other words, the spectral image of the hidden structures is integrated with the three-dimensional representation of tissue 2703.

[0293] The surgical visualization system 2600 is configured to determine one or more distances to key structures. For example, the proximity of the surgical devices 2702a, 2702b relative to the clamp 2712 can be tracked. The surgical visualization system 2700 can utilize time-of-flight distance measurements to determine various distances from the camera 2720 (e.g., utilizing a target wavelength to the surgical device and / or anatomical structure, as further described herein). For example, a triangulation algorithm can then determine the relative distances between structures.

[0294] like Figure 50 As shown, clamp 2712 is positioned between the relative jaws of linear stapler 2702a. In this case, clamp 2712 can be located in the proximity area of ​​linear stapler 2702a, and this proximity area can be limited by the minimum distance between linear stapler 2702a and clamp 2712. In various cases, the proximity area can depend on the steps of surgical device and / or surgical operation. Because clamp 2712 is in the proximity area of ​​linear stapler 2702a, display 2750 is configured to be able to provide warning or alert to clinician. For example, warning can be transmitted as exclamation mark or other symbol on the screen and / or utilize flashing light, light and / or sound. In some cases, warning can be transmitted for example along proximity spectrum indicator mark 2756.

[0295] Similarly, the surgical visualization system 2700 can be configured to detect an additional fastener, such as another clamp, tack, or staple, and track the fastener relative to the surgical device or portion thereof, such as an end effector or shaft of the surgical device. Figure 52 , shows a laparoscopic hernia repair procedure. During such surgery, surgical tacks 2812 can be used to secure surgical mesh 2814 to tissue 2803, such as the abdominal wall. For example, surgical mesh 2814 can be secured to the abdominal wall to prevent bowel from protruding into the abdominal wall.

[0296] In various circumstances, the surgical visualization system can be configured to identify the surgical tack 2812 to ensure that the tack 2812 does not damage critical structures and / or is not positioned within a critical proximity area relative to critical structures. Figure 52 , the surgical visualization system can be configured to identify surgical tacks 2812, mesh 2814, surgical devices, and one or more anatomical structures, such as a hernia 2801 on the intestine. As described herein, the surgical visualization system can utilize spectral imaging to identify one or more structures in combination with structured light to generate a three-dimensional representation of tissue (such as the intestine within the abdominal cavity). In some cases, the system can provide depth confirmation of the tack 2812 via the hyperspectral signature differences of the tack 2812, tissue 2803, hernia 2801, and mesh 2814. In some cases, the surgical visualization system can provide, for example, visibility of a hidden mesh 2814 for engagement with tissue 2803 via the tack 2812. Various surgical visualization systems disclosed herein can identify hernia tacks and, in various cases, notify the clinician when a surgical tack is positioned too close to a surgical device and / or critical structure.

[0297] Exemplary clinical applications

[0298] The various surgical visualization systems disclosed herein may be used in one or more of the following clinical applications.The following clinical applications are non-exhaustive and merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0299] The surgical visualization system disclosed herein can be used in a variety of different types of surgeries, for example, in different medical specialties such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, bariatrics / gastroenterology, and hepato-pancreatico-biliary surgery (HPB). For example, in urological surgery (such as a prostatectomy), ureters can be detected in fat, or connective tissue and / or nerves can be detected in fat. For example, in gynecological oncology surgery (such as a hysterectomy), and in colorectal surgery (such as a low anterior resection (LAR) surgery), ureters can be detected in fat and / or connective tissue. For example, in thoracic surgery (such as a lobectomy), blood vessels can be detected in the lungs or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In bariatric surgery, blood vessels can be detected in fat. For example, in HPB surgery (such as hepatectomy or pancreatectomy), blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0300] In one example, a clinician may want to remove an endometrial fibroid. Based on a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may want to know during surgery which tissues constitute part of the intestine and which tissues constitute part of the rectum. In such a case, a surgical visualization system as disclosed herein can indicate the different types of tissue (intestine vs. rectum) and convey this information to the clinician via the imaging system. In addition, the imaging system can determine the proximity of the surgical device to the selected tissue and communicate this proximity. In such a case, the surgical visualization system can provide increased surgical efficiency without serious complications.

[0301] In another example, a clinician (e.g., a gynecologist) may stay away from certain anatomical areas to avoid getting too close to critical structures, and therefore, the clinician may not be able to remove, for example, all of the endometriosis. A surgical visualization system as disclosed herein may enable a gynecologist to reduce the risk of getting too close to critical structures, allowing the gynecologist to get close enough with a surgical device to remove all of the endometriosis, which may improve patient outcomes (democratizing surgery). Such a system may enable a surgeon to "keep moving" during a surgical procedure rather than repeatedly stopping and restarting in order to identify areas to avoid, for example, particularly during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and given the presence and / or thickness of the tissue involved, the system may be particularly useful for hysterectomy and endometriosis surgery.

[0302] In another example, a clinician may risk dissecting a vessel too close together, and thus potentially affecting the blood supply to lobes other than the target lobe. Furthermore, anatomical variations from patient to patient may result in dissecting vessels (e.g., branches) that affect different lobes depending on the patient. A surgical visualization system as disclosed herein can enable identification of the correct vessel at the desired location, enabling the clinician to perform the dissection with appropriate anatomical certainty. For example, the system can confirm that the correct vessel is in the correct location, and the clinician can then safely divide the vessel.

[0303] In another example, due to uncertainty in the anatomy of a vessel, a clinician may perform multiple dissections before reaching the optimal location. However, it is desirable to perform the dissection at the optimal location first, as more dissections can increase the risk of bleeding. A surgical visualization system, as disclosed herein, can minimize the number of dissections by indicating the correct vessel and optimal location for dissection. For example, the ureters and cardinal ligaments are densely packed and present unique challenges during dissection. In such cases, minimizing the number of dissections may be particularly desirable.

[0304] In another example, a clinician (e.g., a surgical oncologist removing cancerous tissue) may wish to know the identification of key structures, the location of the cancer, the stage of the cancer, and / or an assessment of tissue health. Such information goes beyond what the clinician can see with the "naked eye." A surgical visualization system as disclosed herein can determine such information intraoperatively and / or communicate such information to the clinician to enhance intraoperative decision-making and improve surgical outcomes. In some cases, the surgical visualization system can be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, for example, using an endoscope or an exoscope.

[0305] In another example, a clinician (e.g., a surgical oncologist) may want to turn off one or more alerts about the proximity of surgical tools to one or more critical structures to avoid being too conservative during surgery. In other cases, the clinician may want to receive certain types of alerts such as tactile feedback (e.g., vibration / beep) to indicate proximity and / or a "no-fly zone" to keep sufficiently away from one or more critical structures. For example, a surgical visualization system as disclosed herein can provide flexibility based on the clinician's experience and / or the desired aggressiveness of the surgery. In such cases, the system provides a balance between "knowing too much" and "knowing enough" to anticipate and avoid critical structures. The surgical visualization system can help plan the next steps during surgery.

[0306] Example

[0307] Various aspects of the subject matter described herein are set forth in the following numbered examples.

[0308] Embodiment 1 - A surgical visualization system comprises: a display screen; a surgical device configured to emit a structured light pattern onto a surface; an image sensor configured to identify a structure embedded beneath the surface; and control circuitry in signal communication with the image sensor. The control circuitry is configured to: receive imaging data indicating the structured light pattern on the surface; generate a three-dimensional digital representation of the surface based on the imaging data, obtain an image of the structure and the surgical device from the image sensor; overlay the image of the structure and the surgical device with the three-dimensional digital representation of the surface on the display screen; and determine a distance from the surgical device to the structure based on the image.

[0309] Example 2 - The surgical visualization system according to Example 1 further includes an emitter configured to emit spectral light of multiple wavelengths that can penetrate the surface and reach the structure, wherein the image sensor is configured to detect the reflected spectral light, and wherein the control circuit is further configured to identify the orientation of the structure below the surface based on the reflected spectral light.

[0310] Example 3 - The surgical visualization system of Example 1 or 2, further comprising a three-dimensional camera having the image sensor, and wherein the image comprises a three-dimensional image.

[0311] Example 4 - A surgical visualization system according to Example 1, 2 or 3, wherein the display screen includes a digital proximity spectrum, and wherein the control circuit is further configured to be capable of displaying the distance from the surgical device to the structure on the digital proximity spectrum.

[0312] Example 5 - A surgical visualization system according to Example 4, wherein the digital proximity spectrum includes multiple colors.

[0313] Example 6 - A surgical visualization system according to Example 4, wherein the digital proximity spectrum includes a range of numerical values.

[0314] Example 7 - A surgical visualization system according to Example 4, wherein the digital proximity spectrum includes a plurality of cross-hatched patterns corresponding to a range of distances.

[0315] Example 8 - A surgical visualization system according to Example 1, 2, 3, 4, 5, 6 or 7, wherein the three-dimensional digital representation of the surface and the orientation of the structure are updated in real time on the display screen.

[0316] Example 9 - The surgical visualization system according to Example 1, 2, 3, 4, 5, 6, 7 or 8 also includes a robotic control unit that communicates with the control circuit signal, wherein the surgical device is operably controlled by the robotic control unit, and wherein the robotic control unit is configured to adjust the operation of the surgical device when the distance from the surgical device to the structure decreases to less than a minimum distance.

[0317] Example 10 - The surgical visualization system of Examples 1, 2, 3, 4, 5, 6, 7, 8 or 9, further comprising a contrast agent in the structure, wherein the contrast agent is configured to illuminate the structure, and wherein the image sensor is configured to detect visible light reflected from the illuminated structure.

[0318] Example 11 - The surgical visualization system of Example 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10, further comprising a second surgical device. The control circuitry is further configured to determine a second distance from the second surgical device to the structure based on the image and provide the second distance to the imaging system.

[0319] Example 12 - A surgical visualization system according to Example 11, wherein the display screen is further configured to display the second surgical device and the second distance on a second proximity spectrum indicator.

[0320] Example 13 - The surgical visualization system of Example 11 or 12, wherein the control circuitry is further configured to display a first alert when the distance from the surgical device to the structure decreases to less than a first minimum distance, and to display a second alert when the second distance from the second surgical device to the structure decreases to less than a second minimum distance, the second minimum distance being different from the first minimum distance.

[0321] Example 14 - A surgical visualization system according to Example 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 or 13, wherein the control circuit is configured to triangulate the distance based on the known orientation of the surgical device and the image sensor.

[0322] Embodiment 15 - A surgical visualization system comprising a processor and a memory communicatively coupled to the processor. The memory stores instructions executable by the processor to: receive imaging data indicating a structured light pattern on a surface; generate a three-dimensional digital representation of the surface based on the imaging data; obtain an image of an embedded structure and a surgical device from an image sensor; overlay the image of the embedded structure and the surgical device with the three-dimensional digital representation of the surface on a display screen; and determine a distance from the surgical device to a portion of the surface that covers the embedded structure.

[0323] Example 16 - A surgical visualization system according to Example 15, wherein the orientation of the embedded structure is identified using reflected spectral light that can penetrate the surface and reach the embedded structure.

[0324] Example 17 - A non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a machine to: receive imaging data indicating a structured light pattern on a surface; generate a three-dimensional digital representation of the surface based on the imaging data; obtain a three-dimensional image of an embedded structure and a surgical device from an image sensor; superimpose the image of the embedded structure and the surgical device with the three-dimensional digital representation of the surface on a display screen; and determine a distance from the surgical device to the embedded structure based on the three-dimensional image.

[0325] Example 18 - A non-transitory computer-readable medium according to Example 17, wherein the computer-readable instructions, when executed, further cause the machine to provide a signal indicating the distance to the display screen and to issue a warning signal when the distance meets a predefined threshold distance.

[0326] Example 19 - A surgical visualization system comprises a display and a first robotic tool having a three-dimensional camera. The three-dimensional camera comprises an image sensor. The surgical visualization system further comprises a second robotic tool comprising a spectral light emitter configured to emit spectral light of multiple wavelengths capable of penetrating a surface and reaching structures beneath the surface. The image sensor is configured to detect reflected visible light and the reflected spectral light of the multiple wavelengths. The surgical visualization system further comprises a control circuit in signal communication with the image sensor and the display. The control circuit is configured to: obtain a three-dimensional image of the structure and the second robotic tool from the image sensor; determine a distance from the second robotic tool to the structure based on the three-dimensional image; and provide a signal indicating the distance to the display.

[0327] Example 20 - The surgical visualization system of Example 19, wherein the second robotic tool further comprises a structured light emitter. The control circuitry is further configured to: receive imaging data indicating a structured light pattern on a surface; generate a three-dimensional digital representation of the surface based on the imaging data; and provide a video signal to the display, wherein the three-dimensional image of the structure is integrated with the three-dimensional digital representation of the surface.

[0328] Example 21 - A surgical visualization system comprising: a transmitter configured to transmit a plurality of tissue-penetrating waveforms; a receiver configured to detect the plurality of tissue-penetrating waveforms; an imaging system including a display; and control circuitry in signal communication with the receiver. The control circuitry is configured to receive data representing an image of a hidden portion of a surgical device from the receiver and provide the image of the hidden portion of the surgical device to the display.

[0329] Example 22 - The surgical visualization system according to Example 21 also includes a hyperspectral camera, wherein the hyperspectral camera includes the transmitter and the receiver.

[0330] Example 23 - The surgical visualization system of Example 21 or 22, further comprising a tissue surface mapping system having a structured light source. The control circuitry is further configured to: receive data representing a three-dimensional representation of a tissue surface from the tissue surface mapping system; provide the three-dimensional representation of the tissue surface to the display; and superimpose the image of the hidden portion of the surgical device on the three-dimensional representation of the tissue surface on the display.

[0331] Example 24 - A surgical visualization system according to Example 21, 22 or 23, wherein the surgical device includes a robotic surgical tool.

[0332] Example 25 - A surgical visualization system according to Example 21, 22 or 23, wherein the surgical device includes an aspiration needle.

[0333] Example 26 - A surgical visualization system according to Example 21, 22, 23, 24 or 25, wherein the multiple tissue penetrating waveforms include: a first waveform, which is configured to target the hidden portion of the surgical device; and a second waveform, which is configured to target an anatomical structure.

[0334] Example 27 - A surgical visualization system according to Example 26, wherein the control circuit is further configured to be able to identify a first spectral signature corresponding to the hidden portion of the surgical device and a second spectral signature corresponding to the anatomical structure.

[0335] Example 28 - A surgical visualization system according to Example 26 or 27, wherein the control circuit is further configured to determine the distance between the hidden portion of the surgical device and the anatomical structure.

[0336] Example 29 - A surgical visualization system according to Example 28, wherein the display is configured to convey the distance between the hidden portion of the surgical device and the anatomical structure.

[0337] Example 30 - A surgical visualization system according to Example 28 or 29, wherein the control circuit is further configured to issue an alarm when the distance between the hidden portion of the surgical device and the anatomical structure reaches a threshold minimum distance.

[0338] Example 31 - The surgical visualization system of Example 21, 22, 23, 24, 25, 26, 27, 28, 29, or 30, wherein the control circuit comprises a processor and a memory communicatively coupled to the processor. The memory stores instructions executable by the processor to: receive data representing the image of the hidden portion of the surgical device from the receiver; and provide the image of the hidden portion of the surgical device to the display.

[0339] Example 32 - A surgical visualization system comprising a hyperspectral camera and a control circuit in signal communication with the hyperspectral camera. The hyperspectral camera comprises an emitter and an image sensor. The emitter is configured to emit a plurality of tissue penetration waveforms. The image sensor is configured to detect the plurality of tissue penetration waveforms. The control circuit is configured to: receive data indicating the position of a first critical structure from the plurality of tissue penetration waveforms detected by the image sensor; receive data indicating the position of a second critical structure from the plurality of tissue penetration waveforms detected by the image sensor; and determine a distance between the first critical structure and the second critical structure.

[0340] Example 33 - A surgical visualization system according to Example 32, wherein the first key structure includes one of a surgical device and an anatomical structure.

[0341] Example 34 - A surgical visualization system according to Example 32 or 33, wherein the second critical structure includes one of a surgical device and an anatomical structure.

[0342] Example 35 - The surgical visualization system of Example 32, 33, or 34, wherein the control circuit comprises a processor and a memory communicatively coupled to the processor. The memory stores instructions executable by the processor to: identify the first key structure from the plurality of tissue penetration waveforms detected by the image sensor; identify the second key structure from the plurality of tissue penetration waveforms detected by the image sensor; and determine the distance between the first key structure and the second key structure.

[0343] Example 36 - The surgical visualization system according to Example 32, 33, 34 or 35 also includes a video monitor, wherein the control circuit is further configured to be able to schematically depict the first key structure and the second key structure in real time on the video monitor.

[0344] Embodiment 37 - The surgical visualization system of Embodiment 32, 33, 34, 35, or 36, wherein the emitter is further configured to emit a structured light pattern configured to reach a surface, the image sensor is further configured to detect the structured light pattern, and the control circuitry is further configured to receive data representing a three-dimensional representation of the surface from the structured light pattern detected by the image sensor.

[0345] Example 38 - A surgical visualization system according to Example 37, wherein the control circuit is further configured to generate an image based on the three-dimensional representation of the surface and superimpose the schematic depiction of the first key structure and the second key structure on the image.

[0346] Example 39 - A non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a machine to: receive data representing a first image of a first hidden structure from an image sensor; provide the first image of the first hidden structure to a display; receive data representing a second image of a second hidden structure from the image sensor; provide the second image of the second hidden structure to the display; and determine a distance between the first hidden structure and the second hidden structure.

[0347] Example 40 - A non-transitory computer-readable medium according to Example 39, wherein the computer-readable instructions, when executed, further cause the machine to: receive data from a receiver configured to detect a structured light pattern on a surface; generate a three-dimensional rendering of the surface based on the data; provide the three-dimensional rendering of the surface to the display; and superimpose the first image of the first hidden structure and the second image of the second hidden structure on the three-dimensional rendering of the surface on the display.

[0348] Example 41 - A surgical visualization system comprising a first projector, a second projector, and a control circuit. The first projector is configured to emit a structured light pattern onto a surface of an anatomical structure. The second projector is configured to emit spectral light of multiple wavelengths that can penetrate the anatomical structure and reach the nail line. The control circuit communicates with an image sensor signal. The control circuit is configured to: receive structured light data indicating the structured light pattern on the surface of the anatomical structure from the image sensor; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data indicating a spectral image of the nail line from the image sensor; generate the spectral image of the nail line based on the spectral light data; and determine a distance relative to the nail line.

[0349] Example 42 - The surgical visualization system of Example 41 further comprises a video monitor, wherein the control circuit is in signal communication with the video monitor. The control circuit is further configured to selectively provide a first video signal to the video monitor, the first video signal indicating the three-dimensional representation of the anatomical structure in real time. The control circuit is further configured to selectively provide a second video signal to the video monitor, the second video signal indicating the position of the staple line in real time.

[0350] Example 43 - A surgical visualization system according to Example 42, wherein the control circuit is further configured to selectively integrate the first video signal and the second video signal to generate a video depicting the orientation of the staple line superimposed on the three-dimensional representation of the anatomical structure.

[0351] Example 44 - A surgical visualization system according to Example 42 or 43, wherein the control circuit is further configured to selectively provide a first signal indicating the distance relative to the staple line to the video monitor.

[0352] Example 45 - The surgical visualization system of Example 41, 42, 43, or 44, wherein the control circuitry is further configured to receive spectral light data indicative of a spectral image of a surgical end effector of a robotic tool from the image sensor, and the control circuitry is further configured to generate the spectral image of the surgical end effector from the spectral light data.

[0353] Example 46 - A surgical visualization system according to Example 45, wherein the control circuit is further configured to triangulate the distance between the surgical end effector and the staple line based on the coordinates of the image sensor and the robotic tool.

[0354] Example 47 - The surgical visualization system of Example 41, 42, 43, or 44, wherein the control circuit is further configured to receive spectral light data indicative of a spectral image of a trocar of a circular stapler from the image sensor, and the control circuit is further configured to generate the spectral image of the trocar based on the spectral light data.

[0355] Example 48 - The surgical visualization system of Example 47, wherein the control circuit is further configured to receive spectral light data indicative of a spectral image of a circular stapler anvil from the image sensor, and the control circuit is further configured to generate the spectral image of the circular stapler anvil based on the spectral light data.

[0356] Example 49 - A surgical visualization system according to Example 48, wherein the control circuit is further configured to determine the distance between the cannula needle and the circular stapler anvil.

[0357] Example 50 - The surgical visualization system of Example 49, wherein the control circuit is further configured to provide a first signal to the video monitor indicating the distance between the circular stapler anvil and the staple line. The control circuit is further configured to provide a second signal to the video monitor indicating the distance between the trocar and the circular stapler anvil.

[0358] Example 51 - A surgical visualization system according to Example 42 or 43, wherein the control circuit is further configured to provide the second video signal to the video monitor to track the staple line based on user selection input.

[0359] Example 52 - A surgical visualization system according to Example 41, 42, 43, 44, 45, 46, 47, 48, 49, 50 or 51, wherein the control circuit communicates signals with a robotic control unit, the robotic control unit being operably configured to control a robotic arm supporting a surgical device, and wherein the robotic control unit is operably configured to provide a control signal to move the robotic arm toward a portion of the anatomical structure that obscures the staple line from view.

[0360] Example 53 - The surgical visualization system according to Example 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51 or 52 also includes a hyperspectral camera, which includes the first projector, the second projector and the image sensor.

[0361] Embodiment 54 - A surgical visualization system comprising a processor and a memory communicatively coupled to the processor. The memory stores instructions that, when executed by the processor, receive structured light data from an image sensor indicating a structured light pattern on a surface of an anatomical structure; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data from the image sensor indicating a spectral image of a staple line; generate a spectral image of the staple line based on the spectral light data; and determine a distance relative to the staple line.

[0362] Example 55 - A surgical visualization system according to Example 54, wherein the memory stores instructions that, when executed by the processor: selectively provide a first video signal to a video monitor that indicates the three-dimensional representation of the anatomical structure in real time; and selectively provide a second video signal to the video monitor that indicates the orientation of the staple line in real time.

[0363] Example 56 - A surgical visualization system according to Example 55, wherein the memory stores instructions that, when executed by the processor, selectively integrate the first video signal and the second video signal to generate a video depicting the orientation of the staple line superimposed on the three-dimensional representation of the anatomical structure.

[0364] Example 57 - A surgical visualization system according to Example 55 or 56, wherein the memory stores instructions that, when executed by the processor, selectively provide a first signal to the video monitor indicating the distance relative to the staple line.

[0365] Example 58 - A non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a machine to: receive structured light data from an image sensor indicating a structured light pattern on a surface of an anatomical structure; calculate a three-dimensional representation of the anatomical structure based on the structured light data; receive spectral light data from the image sensor indicating a spectral image of a nail line; generate a spectral image of the nail line based on the spectral light data; and determine a distance relative to the nail line.

[0366] Example 59 - A non-transitory computer-readable medium storing computer-readable instructions according to Example 58, wherein the computer-readable instructions, when executed, further cause the machine to: selectively provide a first video signal to a video monitor indicating the three-dimensional representation of the anatomical structure in real time; and selectively provide a second video signal to the video monitor indicating the orientation of the nail line in real time.

[0367] Example 60 - A non-transitory computer-readable medium storing computer-readable instructions according to Example 59, wherein the computer-readable instructions, when executed, further cause the machine to selectively integrate the first video signal and the second video signal to generate a video depicting the orientation of the staple line superimposed on the three-dimensional representation of the anatomical structure.

[0368] Although multiple forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such details. Without departing from the scope of this disclosure, many modifications, variations, changes, substitutions, combinations and equivalents to these forms may be realized, and those skilled in the art will appreciate many modifications, variations, changes, substitutions, combinations and equivalents to these forms. In addition, alternatively, the structure of each element associated with the described form may be described as a device for providing the function performed by the element. In addition, where materials for certain components are disclosed, other materials may also be used. Therefore, it should be understood that the above-mentioned specific embodiments and the appended claims are intended to encompass all such modifications, combinations and variations within the scope of the forms disclosed by the present invention. The appended claims are intended to encompass all such modifications, variations, changes, substitutions, modifications and equivalents.

[0369] The above detailed description has been described using block diagrams, flow charts, and / or examples to illustrate various forms of apparatus and / or methods. As long as such block diagrams, flow charts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flow charts, and / or examples can be implemented individually and / or collectively by a variety of hardware, software, firmware, or any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented in whole or in part in an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as any combination thereof, and that designing circuit systems and / or writing software and / or hardware code will be within the skill of those skilled in the art based on the present disclosure. In addition, those skilled in the art will recognize that the mechanisms of the subject matter described herein can be distributed as one or more program products in a variety of forms, and that the illustrative forms of the subject matter described herein are applicable regardless of the specific type of signal-bearing medium used for actual distribution.

[0370] Instructions for programming logic to perform various disclosed aspects may be stored in a memory in the system, such as dynamic random access memory (DRAM), cache, flash memory, or other memory. In addition, instructions may be distributed via a network or through other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but is not limited to a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), and a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, a flash memory, or a tangible, machine-readable storage device used when transmitting information over the Internet via an electrical signal, an optical signal, an acoustic signal, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.). Thus, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0371] As used in any aspect of this document, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more separate instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), a field programmable gate array (FPGA)), state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, or the like. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application-specific integrated circuit, electronic circuitry forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program to at least partially implement the methods and / or apparatus described herein, or a microprocessor configured by a computer program to at least partially implement the methods and / or apparatus described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and / or electronic circuitry forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). Those skilled in the art will recognize that the subject matter described herein can be implemented in analog or digital form, or some combination thereof.

[0372] As used in any aspect of this document, the term "logic" may refer to an application, software, firmware, and / or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, instruction sets, and / or data hard-coded (e.g., non-volatile) in a memory device.

[0373] As used in any aspect herein, the terms "component," "system," "module," and the like may refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0374] As used in any aspect herein, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, where a "step" refers to manipulations of physical quantities and / or logical states, which may (but need not) take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. The terms "input" and "output" are commonly used to refer to such signals as bits, values, elements, symbols, characters, terms, numbers, and the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to such quantities and / or states.

[0375] The network may include a packet-switched network. The communication devices may be capable of communicating with each other using a selected packet-switched network communication protocol. An exemplary communication protocol may include an Ethernet communication protocol that may allow communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" published by the Institute of Electrical and Electronics Engineers (IEEE) in December 2008 and / or later versions of this standard. Alternatively or additionally, the communication devices may be capable of communicating with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may be capable of communicating with each other using the Frame Relay communication protocol. The Frame Relay communication protocol may conform to or be compatible with standards published by the International Telegraph and Telephone Consultative Committee (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard entitled "ATM-MPLS Network Interworking 2.0" published by the ATM Forum in August 2001 and / or later versions thereof. Of course, different and / or later developed connection-oriented network communication protocols are also contemplated herein.

[0376] Unless otherwise expressly indicated in the above disclosure, it is understood that discussions in the above disclosure using terms such as "process," "compute," "calculate," "determine," and "display" refer to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities within the computer system's registers and memories and transform them into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0377] One or more components may be referred to herein as being "configured to be able to," "configurable to be able to," "operable / operably," "suitable / adaptable to," "able to," "conformable / conform to," etc. Those skilled in the art will recognize that, unless the context indicates otherwise, "configured to be able to" may generally encompass components in an active state and / or components in an inactive state and / or components in a standby state.

[0378] The terms "proximal" and "distal" are used herein relative to a clinician manipulating the handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion positioned away from the clinician. It should also be understood that for brevity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein in conjunction with the accompanying drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0379] Those skilled in the art will recognize that, in general, the terms used herein, and in particular in the appended claims (e.g., the bodies of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "comprising" should be interpreted as "including, but not limited to," etc.). Those skilled in the art will also understand that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation, no such intent is present. For example, to aid understanding, the following appended claims may contain use of the introductory phrases "at least one" and "one or more" to introduce claims. However, the use of such phrases should not be construed as implying that introducing a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); this also applies to the use of definite articles used to introduce claim recitations.

[0380] In addition, even if a specific number of claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should generally be interpreted to mean at least the recited number (e.g., a bare recitation of "two recitations," without other modifiers, generally means at least two recitations, or two or more recitations). Furthermore, in those instances where a convention similar to "at least one of A, B, and C, etc." is used, generally, such construction is intended to have the meaning that one skilled in the art would understand the convention (e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention similar to "at least one of A, B, or C, etc." is used, generally, such construction is intended to have the meaning that one skilled in the art would understand the convention (e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art will also understand that, generally, unless the context indicates otherwise, transitional words and / or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to encompass the possibility of including one, either, or both of the terms. For example, the phrase "A or B" will generally be understood to include the possibility of "A" or "B" or "A and B."

[0381] With respect to the appended claims, those skilled in the art will understand that the operations described therein may generally be performed in any order. In addition, although various operational flow charts are presented in one or more sequences, it will be understood that the various operations may be performed in an order other than the order shown, or the various operations may be performed simultaneously. Unless the context dictates otherwise, examples of such alternative orderings may include overlapping, interleaved, interrupted, reordered, incremental, preliminary, supplemental, simultaneous, reverse, or other altered orderings. Furthermore, unless the context dictates otherwise, terms such as "responsive to," "related to," or other past tense adjectives are generally not intended to exclude such variations.

[0382] It is worth noting that any reference to "one aspect," "an aspect," "an example," or "an example" means that the specific features, structures, or characteristics described in connection with the aspect are included in at least one aspect. Therefore, the phrases "in one aspect," "in an aspect," "in an example," and "in an example" appearing in various places throughout this specification do not necessarily refer to the same aspect. Furthermore, specific features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0383] Any patent application, patent, non-patent publication, or other disclosure material referred to in this specification and / or listed in any Application Data Sheet is incorporated herein by reference to the extent that the incorporated material is inconsistent herein. Therefore, and to the extent necessary, the disclosure explicitly set forth herein supersedes any conflicting material incorporated herein by reference. Any material, or portion thereof, allegedly incorporated herein by reference that conflicts with existing definitions, statements, or other disclosure material listed herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing disclosure material.

[0384] In summary, the numerous benefits resulting from the use of the concepts described herein have been described. For purposes of illustration and description, one or more specific embodiments have been provided above. These specific embodiments are not intended to be exhaustive or to limit the invention to the precise forms disclosed. The invention may be modified or varied in light of the above teachings. The form or forms selected and described are intended to illustrate the principles and practical applications, thereby enabling one of ordinary skill in the art to utilize various forms and modifications as appropriate for the particular use contemplated. The claims submitted herewith are intended to define the full scope.

Claims

1. A surgical visualization system comprising: Display screen; a surgical device configured to emit a structured light pattern onto a surface; an image sensor configured to identify structures embedded beneath the surface; as well as a control circuit in signal communication with the image sensor, wherein the control circuit is configured to: receiving imaging data indicative of the structured light pattern on the surface; generating a three-dimensional digital representation of the surface based on the imaging data; obtaining an image of the structure and the surgical device from the image sensor, the image sensor being configured to detect reflected spectral light that can penetrate the surface and reach the structure; superimposing the images of the structure and the surgical device with the three-dimensional digital representation of the surface on the display screen; identifying a location of the structure beneath the surface based on the reflected spectral light in the image of the structure; and A distance from the surgical device to the structure is determined based on the image, and an alert can be provided to the clinician at a first distance if the temperature of the surgical device is above a predefined threshold, and an alert can be provided to the clinician at a second distance if the temperature of the surgical device is less than or equal to the predefined threshold, wherein the second distance is less than the first distance.

2. The surgical visualization system of claim 1 , further comprising an emitter configured to emit spectral light at multiple wavelengths.

3. The surgical visualization system of claim 2, further comprising a three-dimensional camera having the image sensor, and wherein the image comprises a three-dimensional image.

4. The surgical visualization system of claim 3, wherein: The display screen includes a digital proximity spectrum, and wherein the control circuit is further configured to display the distance from the surgical device to the structure on the digital proximity spectrum.

5. The surgical visualization system of claim 4, wherein: The digital proximity spectrum includes a plurality of colors.

6. The surgical visualization system of claim 4, wherein: The digital proximity spectrum includes a range of values.

7. The surgical visualization system of claim 4, wherein: The digital proximity spectrum includes a plurality of cross-hatched patterns corresponding to a range of distances.

8. The surgical visualization system of claim 1 , wherein: The three-dimensional digital representation of the surface and the position of the structure are updated in real time on the display screen.

9. The surgical visualization system of claim 1 , further comprising a robotic control unit in signal communication with the control circuit, wherein the surgical device is operably controlled by the robotic control unit, and wherein the robotic control unit is configured to adjust operation of the surgical device when the distance from the surgical device to the structure decreases to less than a minimum distance.

10. The surgical visualization system of claim 1, further comprising a contrast agent in the structure, wherein the contrast agent is configured to illuminate the structure, and wherein the image sensor is configured to detect visible light reflected from the illuminated structure.

11. The surgical visualization system of claim 1 , wherein: The control circuit is configured to triangulate the distance based on known positions of the surgical device and the image sensor.

12. A non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a machine to: receiving imaging data indicative of a structured light pattern on a surface; generating a three-dimensional digital representation of the surface based on the imaging data; obtaining a three-dimensional image of the embedded structure and the surgical device from an image sensor configured to detect reflected light in a spectrum that can penetrate the surface and reach the structure; superimposing the images of the embedded structure and the surgical device with the three-dimensional digital representation of the surface on a display screen; identifying a location of the structure beneath the surface based on the reflected spectral light in the image of the structure; and A distance from the surgical device to the embedded structure is determined based on the three-dimensional image, and an alert can be provided to the clinician at a first distance if the temperature of the surgical device is above a predefined threshold, and an alert can be provided to the clinician at a second distance if the temperature of the surgical device is less than or equal to the predefined threshold, wherein the second distance is less than the first distance.

13. The non-transitory computer readable medium of claim 12, wherein: The computer readable instructions, when executed, further cause the machine to: providing a signal indicative of the distance to the display screen; and A warning signal is issued when the distance meets a predefined threshold distance.

Images