Surgical system with devices for both intraluminal and extraluminal access

Abstract

The surgical system (110, 700, 702, 802, 810, 10100, 10200, 10400, 10500) comprises a first scoping device (10102, 10228, 10412, 10502) having a first portion configured to be inserted and positioned within an extraluminal anatomical space and a second portion distal to the first portion and configured to be positioned within an endoluminal anatomical space, and a second instrument (10462) configured to be inserted into the extraluminal anatomical space and configured to couple to and move the first portion of the first scoping device within the extraluminal anatomical space to facilitate movement of the second portion of the first scoping device while the second portion is positioned within the endoluminal anatomical space.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 249,980, entitled “Cooperative Access,” filed on 29 September 2021, the entire disclosure of which is incorporated herein by reference.

[0002] (Field of invention) The present invention generally relates to surgical systems and methods for using the same, such as mooring, coordinated endoscopic and laparoscopic access, and tissue manipulation. [Background technology]

[0003] Surgical systems often incorporate imaging systems that allow clinicians to view the surgical site and / or one or more parts thereof on one or more displays (e.g., monitors, computer tablet screens, etc.). These displays may be localized to the surgical theater and / or remote. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display visible to the clinician(s).

[0004] Imaging systems may be limited by the information they can recognize and / or communicate to a clinician(s). For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. In another example, certain imaging systems may not be able to communicate and / or transmit certain information to a clinician(s) during surgery. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, improved electric surgical tools are still needed.

Means for Solving the Problem

[0006] A surgical system is provided. In an exemplary embodiment, the surgical system includes a first portion configured to be inserted and positioned within an extra-luminal anatomical space, and a second portion distal to the first portion and configured to be positioned within the intra-luminal anatomical space, a first scope device having; and a second instrument configured to be inserted within the extra-luminal anatomical space and coupled to the first portion of the first scope device within the extra-luminal anatomical space to move the first portion in order to facilitate movement of the second portion of the first scope device while the second portion is disposed within the intra-luminal anatomical space. The first scope device is a flexible body having a working channel extending therethrough and a first imaging system at a distal end of the first scope device, the working channel being configured to enable the distal end of the first instrument to be inserted into the extra-luminal anatomical space and inserted through the extra-luminal anatomical space into the intra-luminal anatomical space such that the first instrument is present in both the extra-luminal space and the intra-luminal space.

[0007] The second instrument can have various configurations. In some embodiments, the second instrument can be configured to couple to the first portion of the first scope device at a predetermined position within the extra-luminal anatomical space and directly adjacent to a tissue wall defining at least a portion of the intra-luminal anatomical space. In a particular embodiment, the second instrument can include a rigid shaft having an end effector at a distal end. The end effector can be configured to couple to the first portion of the first scope device.

[0008] In some embodiments, the system can include a cannula having a lumen extending therethrough. The cannula can be configured to be disposed within a tissue wall that defines at least a portion of the intraluminal anatomical space, and the distal end of the flexible body can be configured to be inserted from the extraluminal anatomical space through the lumen into the intraluminal anatomical space. In certain embodiments, the second instrument can be further configured to couple to the cannula and move the cannula to facilitate movement of a second portion of the first scope device while the distal end of the flexible body is within the intraluminal anatomical space. In such embodiments, the second instrument can include a rigid shaft having an end effector at the distal end. The end effector can be configured to couple to the cannula.

[0009] In some embodiments, the system can include a fluid port configured to ventilate the extraluminal anatomical space.

[0010] In another exemplary embodiment, the surgical system includes an anchor member disposed within the extraluminal anatomical space and configured to contact a tissue wall that at least partially defines the intraluminal anatomical space, a first portion configured to be inserted and positioned within the extraluminal anatomical space, and a second portion distal to the first portion and configured to be positioned within the intraluminal anatomical space, a cannula having; and a selectively deployable stabilization member disposed on the first portion of the cannula within the extraluminal anatomical space. The cannula is configured such that the distal end of the first instrument is inserted into the extraluminal anatomical space and through the extraluminal anatomical space into the intraluminal anatomical space such that the first instrument is present in both the extraluminal anatomical space and the intraluminal anatomical space. The selectively deployable stabilization member is configured to couple to the anchor member when in a deployed state to provide an anchor point for the first instrument and facilitate pivoting movement of the first instrument within the intraluminal anatomical space.

[0011] The anchor member can have a variety of configurations. In some embodiments, the anchor member may be further configured to seal a portion of the intraluminal anatomical space.

[0012] In some embodiments, the system may further include magnets positioned within the anchor member. The magnets may be configured to connect the selectively deployable stabilizing member to the anchor member when the selectively deployable stabilizing member is in the deployed state.

[0013] In some embodiments, the system may further include a first scope device, which may be configured to be inserted into and through the lumen of a cannula such that a first portion of the scope device is located in the extraluminal anatomical space and a second distal portion of the first portion is located in the intraluminal anatomical space.

[0014] A method is also provided. In one exemplary embodiment, the method includes inserting a first portion of a first scope device into an extraluminal anatomical space, wherein the first scope device has a flexible body having a working channel extending through its interior; inserting a second portion of the first scope device distal to the first portion into an intraluminal anatomical space; inserting the first instrument through the working channel to position the first instrument in both the extraluminal and intraluminal spaces; inserting the second instrument into the extraluminal anatomical space; and moving the second instrument to move the inserted second portion of the first scope device within the intraluminal anatomical space.

[0015] In some embodiments, the method may further include coupling a second instrument to a first portion of a first scope device at a predetermined location within the extraluminal anatomical space and directly adjacent to a tissue wall defining at least a portion of the intraluminal anatomical space.

[0016] In some embodiments, the method may include inserting a cannula through a tissue wall defining at least a portion of an intraluminal anatomical space, wherein the cannula includes a lumen extending through the cannula, and inserting the distal end of a flexible body through the lumen into the intraluminal anatomical space. In such embodiments, the method may include connecting a second instrument to the cannula and moving the cannula to move the second portion of the first scope within the intraluminal anatomical space.

[0017] In some embodiments, the method may include ventilating an extraluminal anatomical space through a fluid port operably coupled to a first part of a first scope device. [Brief explanation of the drawing]

[0018] The present invention will be described with reference to the accompanying drawings. [Figure 1] This is a schematic diagram of one embodiment of a surgical visualization system. [Figure 2] Figure 1 is a schematic diagram of the triangulation between the surgical device, imaging device, and critical structure. [Figure 3] This is a schematic diagram of another embodiment of the surgical visualization system. [Figure 4] This is a schematic diagram of one embodiment of a control system for a surgical visualization system. [Figure 5] This is a schematic diagram of one embodiment of the control circuit of a control system for a surgical visualization system. [Figure 6] This is a schematic diagram of one embodiment of a combination logic circuit for a surgical visualization system. [Figure 7] This is a schematic diagram of one embodiment of the sequential logic circuit of a surgical visualization system. [Figure 8] This is a perspective view of yet another embodiment of the surgical visualization system. [Figure 9] This is a schematic diagram of another embodiment of a control system for a surgical visualization system. [Figure 10]This graph shows the wavelength versus absorption coefficient for various biological substances. [Figure 11] This is a schematic diagram of one embodiment of a spectral emitter for visualizing the surgical site. [Figure 12] This graph illustrates an example hyperspectral identification signature for distinguishing the ureter from an obstruction. [Figure 13] This graph illustrates an example of a hyperspectral identification signature used to distinguish arteries from occluders. [Figure 14] This graph illustrates an example of a hyperspectral identification signature used to distinguish nerves from obstructions. [Figure 15] This is a schematic diagram of one embodiment of a near-infrared (NIR) time-of-flight measurement system used during surgery. [Figure 16] Figure 15 shows the time-of-flight timing diagram for the system. [Figure 17] This is a schematic diagram of another embodiment of a near-infrared (NIR) time-of-flight measurement system used during surgery. [Figure 18] This is a schematic diagram of one embodiment of a computer-implemented bidirectional surgical system. [Figure 19] This is a schematic diagram of one embodiment of a surgical system used for performing surgical procedures in an operating room. [Figure 20] This is a schematic diagram of one embodiment of a surgical system including smart surgical instruments and a surgical hub. [Figure 21] Figure 20 is a flowchart showing how to control smart surgical instruments. [Figure 22] This is a schematic diagram of a conventional surgical system with endoscopic and laparoscopic instruments, showing a partial cross-section of the colon where the endoscopic instrument is inserted into the colon through the natural orifice and the laparoscopic instrument is inserted through the abdominal cavity and interacts with the outer surface of the colon. [Figure 23] This is a schematic diagram of an exemplary embodiment of a surgical system having laparoscopic and intraluminal instruments, showing an intraluminal instrument inserted into the colon through a laparoscopic approach. [Figure 24]This is a schematic diagram of an exemplary embodiment of a surgical system having laparoscopic instruments and two intraluminal instruments, showing the intraluminal instruments inserted into the colon through a laparoscopic approach. [Figure 25] This is a schematic diagram of an exemplary embodiment of a surgical system. [Figure 26] This is a schematic diagram of an exemplary embodiment of a surgical system having laparoscopic and intraluminal instruments, showing an intraluminal instrument inserted into the colon through a laparoscopic approach. [Figure 26a] Figure 26 is a schematic diagram of merged images of the surgical system from a laparoscopic perspective. [Figure 26b] This is a schematic diagram of the merged images of the surgical system shown in Figure 26 from the perspective of an endoscope. [Figure 27] This is a schematic diagram of an exemplary embodiment of a surgical system having laparoscopic and intraluminal instruments, showing an intraluminal instrument inserted into the colon through a laparoscopic approach. [Modes for carrying out the invention]

[0019] To provide a comprehensive understanding of the structure, function, manufacturing and use principles of the devices, systems, and methods disclosed herein, certain exemplary embodiments are described below. One or more examples of these embodiments are illustrated in the accompanying drawings. Those skilled in the art will understand that the devices, systems, and methods described in detail herein and shown in the accompanying drawings are non-limiting exemplary embodiments, and that the scope of the invention is defined solely by the claims. Features illustrated or described in relation to one exemplary embodiment can be combined with features of other embodiments. Such modifications and variations are considered to fall within the scope of the invention.

[0020] Furthermore, in this disclosure, components with similar names in embodiments generally have similar characteristics, and therefore, in a particular embodiment, each characteristic of each component with a similar name is not necessarily described in full detail. In addition, to the extent that linear or circular dimensions are used in the description of the disclosed systems, devices, and methods, such dimensions are not intended to limit the types of shapes that can be used in combination with such systems, devices, and methods. Those skilled in the art will recognize that dimensions equivalent to such linear and circular dimensions can be easily determined for any geometric shape. Those skilled in the art will understand that even if the dimensions are not exact values, they can be considered to be close to them due to various factors such as manufacturing tolerances and the sensitivity of measuring instruments. The size and shape of systems and devices, and their components, may depend at least on the size and shape of the components in which the systems and devices are used.

[0021] surgical visualization Generally, surgical visualization systems are configured to leverage “digital surgery” to acquire additional information about a patient’s anatomical structure and / or surgical procedure. Surgical visualization systems are further configured to transmit data to one or more physicians in a useful format. Various aspects of this disclosure use visualization to provide improved visualization of a patient’s anatomical structure and / or surgical procedure, and / or to provide improved control of surgical tools (also referred herein as “surgical devices” or “surgical instruments”).

[0022] "Digital surgery" may encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, and connectivity both inside and outside the operating room (OR). The various surgical visualization systems described herein can be used in conjunction with robotic surgical systems, but are not limited to use with robotic surgical systems. In certain examples, surgical visualization systems can be generated without a robot, and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be generated without a robot, and / or with limited and / or optional robotic assistance.

[0023] In certain cases, surgical systems incorporating surgical visualization systems can identify critical structures and enable smart incisions to avoid them. Critical structures include anatomical structures such as arteries (e.g., ureters, superior mesenteric arteries), veins (e.g., portal vein), nerves (e.g., phrenic nerve), and / or tumors, among other anatomical structures. In other cases, critical structures may be exogenous structures in an anatomical area, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, plates, and foreign structures. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Smart incisions can, for example, provide improved intraoperative guidance for incisions and / or enable smarter decision-making through techniques for detecting and avoiding critical anatomical structures.

[0024] Surgical systems incorporating surgical visualization systems can enable smart anastomosis, providing more consistent anastomoses at optimal locations through improved workflows. Cancer localization techniques can be improved using surgical visualization platforms. For example, cancer localization methods can identify and track the location, orientation, and margins of a tumor. In certain cases, cancer localization methods can compensate for the movement of surgical instruments, the patient, and / or the patient's anatomical structures during surgical procedures, providing guidance to the physician back to the point of interest.

[0025] Surgical visualization systems can provide improved tissue characterization, as well as lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health status without requiring physical touch, particularly during incision and / or placement of stapling devices within the tissue. Certain tissue characterization techniques can be used without the use of ionizing radiation and / or contrast agents. With regard to lymph node diagnosis and mapping, surgical visualization platforms can, for example, locate, map, and ideally diagnose lymphatic systems and / or lymph nodes involved in cancer diagnosis and staging before surgery.

[0026] During surgical procedures, the information available to the physician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures, such as those embedded or buried within organs, may be at least partially hidden, or even invisible, from the view. Additionally, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or difficult to perceive with the "naked eye." Furthermore, certain structures may move preoperatively (e.g., before the surgical procedure but after the preoperative scan) and / or during the procedure. In such cases, the physician may not be able to accurately determine the location of critical structures during the procedure.

[0027] When the location of critical structures is uncertain, and / or the proximity between critical structures and surgical tools is unknown, the physician's decision-making process may be hindered. For example, a physician may avoid certain areas to avoid inadvertently cutting a critical structure. However, the avoided area may be unnecessarily large and / or at least partially mislocated. Due to uncertainty and / or excessive caution, a physician may fail to reach a specific desired area. For example, even if a critical structure is not in that particular area, and / or the physician's actions in that area would have no adverse effects, over-caution may cause the physician to try to avoid critical structures, leaving behind a portion of the tumor and / or other undesirable tissue. In certain cases, surgical outcomes may improve with increased knowledge and / or certainty, which allows surgeons to be more precise and, in certain cases, more restrained / more aggressive in certain anatomical areas.

[0028] Surgical visualization systems can enable identification and avoidance of critical structures during surgery. Therefore, surgical visualization systems can enhance intraoperative decision-making and improve surgical outcomes. They can provide advanced visualization capabilities beyond what a physician sees with the naked eye, and / or what imaging systems can perceive and / or communicate to the physician. Surgical visualization systems can reinforce and enhance what physicians can know before tissue treatment (e.g., incision), thereby improving outcomes in various cases. As a result, physicians can maintain momentum throughout the surgical procedure, recognizing, for example, that the surgical visualization system is tracking critical structures that may be approached during incision. Surgical visualization systems can provide physicians with sufficient time to pause and / or slow down the surgical procedure and assess the proximity to critical structures to prevent accidental damage to them. A surgical visualization system provides physicians with an ideal, optimized, and / or customizable amount of information, enabling them to move confidently and / or quickly throughout the tissue while avoiding accidental damage to healthy tissue and / or critical structures, thereby minimizing the risk of injury resulting from surgical procedures.

[0029] The surgical visualization system is described in detail below. Generally, the surgical visualization system may include a first optical emitter configured to emit multiple spectral waves, a second optical emitter configured to emit a light pattern, and a receiver or sensor configured to detect visible light, molecular responses to spectral waves (spectroscopic imaging), and / or the light pattern. The surgical visualization system may also include an imaging system and a control circuit that signals the receiver and the imaging system. Based on the output from the receiver, the control circuit can determine distances to the surgical site, such as a geometric surface map of the visible surface at the surgical site, e.g., a three-dimensional surface topography, and distances to at least partially hidden structures. The imaging system can communicate the geometric surface map and distances to the physician. In such an example, the augmented view of the surgical site provided to the physician may provide a display of hidden structures in the context related to the surgical site. For example, the imaging system may virtually enhance hidden structures on the geometric surface map of the hidden and / or obstructed tissue, as well as lines drawn on the ground to indicate utility piping below the surface. Additionally or alternatively, the imaging system can communicate the proximity of a surgical tool to visible obstructing tissue and / or to at least partially hidden structures, and / or the depth of structures hidden below the visible surface of the obstructing tissue. For example, a visualization system can determine the distance to an extended line on the surface of visible tissue and communicate that distance to the imaging system.

[0030] Throughout this disclosure, unless otherwise specified, all references to “light” may include EMR or photons in the visible and / or invisible portions of the electromagnetic radiation (EMR) wavelength spectrum. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by the human eye) and may be referred to as “visible light” or simply “light.” The typical human eye responds to wavelengths in air from approximately 380 nm to approximately 750 nm. The invisible spectrum (e.g., the non-emission spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum. The invisible spectrum is not detectable by the human eye. Wavelengths longer than approximately 750 nm are longer than the red visible spectrum and are invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths shorter than approximately 380 nm are shorter than the violet spectrum and are invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0031] Figure 1 illustrates one embodiment of the surgical visualization system 100. The surgical visualization system 100 is configured to create a visual representation of critical structures 101 within an anatomical region. Critical structures 101 may include one or more critical structures. As discussed herein, critical structures 101 may be any of the following: anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, blood vessels, tumors, or other anatomical structures; or exogenous structures such as surgical devices, surgical fasteners, surgical clips, surgical clasps, bougies, surgical bands, surgical plates, or other exogenous structures. As discussed herein, critical structures 101 may be specific to each patient and / or procedure. Embodiments of critical structures and embodiments of identifying critical structures using the visualization system are further described in U.S. Patent No. 10,792,034, entitled “Visualization of Surgical Devices,” issued on October 6, 2020, which is incorporated herein by reference in its entirety.

[0032] In some examples, the critical structure 101 may be embedded in tissue 103. In other words, tissue 103 can be any of a variety of tissues, such as fat, connective tissue, adhesions, and / or organs. In other words, the critical structure 101 may be located beneath the surface 105 of tissue 103. In such examples, tissue 103 obscures the critical structure 101 from the physician's "gross eye" view. Tissue 103 also obscures the critical structure 101 from the field of view of the imaging device 120 of the surgical visualization system 100. Instead of being completely obscured, the critical structure 101 may be partially obscured from the field of view of the physician and / or the imaging device 120.

[0033] The surgical visualization system 100 can be used in clinical analysis and / or medical intervention. In certain cases, the surgical visualization system 100 can be used intraoperatively to provide a physician with real-time information, such as proximity data, dimensions, and / or distance, during the surgical procedure. Those skilled in the art will recognize that the information is not exactly real-time, but may still be considered real-time due to a variety of reasons, such as time delays caused by data transmission, time delays caused by data processing, and / or the sensitivity of measuring instruments. The surgical visualization system 100 is configured for intraoperative identification of critical structures and / or to facilitate avoidance of critical structures 101 by surgical devices. For example, by identifying critical structures 101, a physician can avoid operating surgical devices around critical structures 101 and / or areas of a predetermined proximity to critical structures 101 during the surgical procedure. In another embodiment, by identifying critical structure 101, the physician can avoid incisions on and / or near critical structure 101, thereby helping to prevent damage to critical structure 101 and / or to surgical devices being damaged by critical structure 101.

[0034] The surgical visualization system 100 is configured to incorporate tissue identification and geometric surface mapping in combination with the surgical visualization system's distance sensor system 104. When combined, these features of the surgical visualization system 100 can determine the location of critical structures 101 within an anatomical region, and / or the proximity of surgical devices 102 to the surface 105 of visible tissue 103 and / or critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system, for example, an imaging device 120 configured to provide a real-time view of the surgical site. The imaging device 120 may include, for example, a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of the image based on molecular responses to various wavelengths. The view from the imaging device 120 can be provided to the physician in real time on a display (e.g., a monitor, computer tablet screen, etc.). The displayed view can be enhanced with additional information based on tissue identification, landscape mapping, and the distance sensor system 104. In such an example, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide the physician with advanced data synthesis and integrated information during surgery.

[0035] The imaging device 120 can be configured to detect, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). Embodiments of the imaging device 120 include scopes, such as endoscopes, arthroscopes, angioscopes, bronchoscopes, common cholangioscopy, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopy (gastroscopy), laryngoscopes, nasopharyngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, or exoscopes. Scopes can be particularly useful in minimally invasive surgical procedures. For open surgery applications, the imaging device 120 may not include a scope.

[0036] The tissue identification subsystem can be achieved using a spectral imaging system. The spectral imaging system may rely on imaging techniques such as hyperspectral imaging, multispectral imaging, or selective spectral imaging. Embodiments of hyperspectral imaging of tissues are further described in U.S. Patent No. 9,274,047, published March 1, 2016, entitled "System and Method for Gross Anatomic Pathology Using Hyperspectral Imaging," which is incorporated herein by reference in its entirety.

[0037] Surface mapping subsystems can be achieved using optical pattern systems. Various surface mapping techniques that use optical patterns (or structured light) for surface mapping can be utilized in the surgical visualization systems described herein. Structured light is the process of projecting a known pattern (often a grid or horizontal bars) onto a surface. In certain cases, invisible (i.e., undetectable) structured light can be utilized, and structured light can be used without interfering with other computer vision tasks where the projected pattern may interfere. For example, interference can be prevented by using infrared light that repeats two exactly opposite patterns or visible light at a very fast frame rate. Embodiments of surgical systems including surface mapping, as well as light sources and projectors for projecting light patterns, are further described in U.S. Patent Application Publication No. 2017 / 0055819, “Set Comprising A Surgical Instrument,” published on 2 March 2017; U.S. Patent Application Publication No. 2017 / 0251900, “Depiction System,” published on 7 September 2017; and U.S. Patent Application No. 16 / 729,751, “Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto,” filed on 30 December 2019, the entirety of which is incorporated herein by reference.

[0038] The distance determination system can be integrated into a surface mapping system. For example, structured light can be used to generate a three-dimensional (3D) virtual model of the visible surface 105, and various distances to the visible surface 105 can be determined. Additionally or alternatively, the distance determination system can rely on time-of-flight measurement to determine one or more distances to specific tissues (or other structures) at the surgical site.

[0039] The surgical visualization system 100 also includes a surgical device 102. The surgical device 102 can be any suitable surgical device. Embodiments of the surgical device 102 include surgical dissection instruments, surgical staplers, surgical grippers, clip applicators, smoke removers, surgical energy devices (e.g., unipolar probes, bipolar probes, ablation probes, ultrasound devices, ultrasound end effectors, etc.). In some embodiments, the surgical device 102 includes an end effector having opposing jaws that extend from the distal end of the shaft of the surgical device 102 and are configured to engage tissue between them.

[0040] The surgical visualization system 100 can be configured to identify critical structures 101 and the proximity of surgical devices 102 to the critical structures 101. The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral camera, a multispectral camera, or a selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral image sensor, detector, and / or three-dimensional camera lens). For example, the imaging device 120 may include right and left lenses used together to simultaneously record two two-dimensional images, thereby generating a three-dimensional (3D) image of the surgical site, rendering the three-dimensional image of the surgical site, and / or determining one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicating the topography of visible tissue, as well as the identification and location of hidden important structures, as further described herein. For example, as shown in Figure 1, the field of view of the imaging device 120 can be superimposed on a light pattern (structured light) on the surface 105 of the tissue 103.

[0041] As in this illustrated embodiment, the surgical visualization system 100 can be incorporated into the surgical system 110. The robotic surgical system 110 can have various configurations, as discussed herein. In the illustrated embodiment, the robotic surgical system 110 includes a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 each include a rigid structural member 116 and a joint 118, which can each include servo motor control. The first robotic arm 112 is configured to operate a surgical device 102, and the second robotic arm 114 is configured to operate an imaging device 120. The robotic control unit of the robotic surgical system 110 is configured to generate control motions to the first robotic arm 112 and the second robotic arm 114, which can act on the surgical device 102 and the imaging device 120, respectively.

[0042] In some embodiments, one or more of the robot arms 112, 114 can be separate from the main robot system 110 used in surgical procedures. For example, at least one of the robot arms 112, 114 can be positioned and aligned to a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robot arms 112, 114 can control and / or align the position of the robot arms 112, 114 to a specific coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned to a specific coordinate system.

[0043] Examples of robotic surgical systems include the Ottava® robot-assisted surgical system (Johnson & Johnson of New Brunswick, NJ), the da Vinci® surgical system (Intuitive Surgical, Inc. of Sunnyvale, CA), the Hugo® robot-assisted surgical system (Medtronic PLC of Minneapolis, MN), the Versius® surgical robot system (CMR Surgical Ltd of Cambridge, UK), and the Monarch® platform (Auris Health, Inc. of Redwood City, CA).Various robotic surgical procedures and embodiments using robotic surgical procedures are described in U.S. Patent Application Publication No. 2018 / 0177556, “Flexible Instrument Insertion Using An Adaptive Force Threshold,” filed on December 28, 2016; U.S. Patent Application Publication No. 2020 / 0000530, “Systems And Techniques For Providing Multiple Perspectives During Medical Procedures,” filed on April 16, 2019; U.S. Patent Application Publication No. 2020 / 0170720, “Image-Based Branch Detection And Mapping For Navigation,” filed on February 7, 2020; U.S. Patent Application Publication No. 2020 / 0188043, “Surgical Robotics System,” filed on December 9, 2019; and “Systems And Methods For Concomitant Medical Further details are provided in U.S. Patent Application Publication No. 2020 / 0085516, entitled “Procedures”, U.S. Patent No. 8,831,782, entitled “Patient-Side Surgeon Interface For A Teleoperated Surgical Instrument,” filed on July 15, 2013, and International Publication No. 2014151621, entitled “Hyperdexterous Surgical System,” filed on March 13, 2014.

[0044] The surgical visualization system 100 also includes an emitter 106. The emitter 106 is configured to emit light patterns such as stripes, grid lines, and / or dots to enable determination of the topography or landscape of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from the emitter 106 located on the surgical device 102 and / or one of the robotic arms 112, 114 and / or the imaging device 120. In one embodiment, the projected light array 130 is employed by the surgical visualization system 100 to determine the shape defined during surgery by the surface 105 of the tissue 103 and / or the motion of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0045] As in this illustrated embodiment, the imaging device 120 may include an optical waveform emitter 123, either mounted on the imaging device 120 or otherwise attached. The optical waveform emitter 123 is configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach critical structures 101. The imaging device 120 and the optical waveform emitter 123 may be positionable by a robotic arm 114. The optical waveform emitter 123 may be mounted on the imaging device 122 or otherwise attached, and in other embodiments, it may be positioned on a surgical device separate from the imaging device 120. A corresponding waveform sensor 122 on the imaging device 120 (e.g., an image sensor, spectrometer, or vibration sensor) is configured to detect the effects of electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted from the optical waveform emitter 123 can be configured to enable the identification of anatomical structures and / or types of body structures, such as critical structures 101. Identification of critical structures 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one embodiment, the wavelength of the electromagnetic radiation 124 can be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other embodiments, the waveform sensor 122 and the optical waveform emitter 123 may include, for example, a photoacoustic imaging system.

[0046] The distance sensor system 104 of the surgical visualization system 100 is configured to determine one or more distances at a surgical site. The distance sensor system 104 can be a time-of-flight distance sensor system including an emitter such as the emitter 106 as in this illustrated embodiment and a receiver 108. In other examples, the time-of-flight emitter may be separate from the structured light emitter. The emitter 106 may include a very small laser source, and the receiver 108 may include a matching sensor. The distance sensor system 104 is configured to detect "time of flight," or the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, the distance sensor system 104 is able to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104.

[0047] In this illustrated embodiment, the receiver 108 of the distance sensor system 104 is positioned on the surgical device 102, but in other embodiments, the receiver 108 may be mounted on a separate surgical device instead of the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar extending through the surgical device 102 to reach the surgical site. In yet another embodiment, the receiver 108 for the distance sensor system 104 may be mounted on a robotic control arm of a robotic system 110 separate from the first robotic arm 112 to which the surgical device 102 is coupled (e.g., on a second robotic arm 114), on a movable arm operated by another robot, or on an operating room (OR) table or fixture. In some embodiments, the imaging device 120 includes a receiver 108 that allows the distance from the emitter 106 on the surgical device 102 to the surface 105 of the tissue 103 to be determined using a line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, based on the known positions of the emitter 106 (on the surgical device 102) and receiver 108 (on the imaging device 120) of the distance sensor system 104, distance de The three-dimensional position of receiver 108 can be determined and / or recorded relative to the robot coordinate plane during surgery.

[0048] As in this illustrated embodiment, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the distance sensor system 104 can be controlled by the second robotic arm 114. In another embodiment, the surgical visualization system 100 may be available separately from the robotic system. In such an example, the distance sensor system 104 may be independent of the robotic system.

[0049] In Figure 1, d e d is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and t d is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue 103. The distance sensor system 104 measures the emitter-tissue distance d e It is configured to determine the device-organizational distance d. t This can be obtained from a known position on the surgical device 102, for example, on its shaft proximal to the distal end of the surgical device 102, relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t is the emitter-tissue distance d e This can be determined from. In some embodiments, the shaft of the surgical device 102 may include one or more articulated joints that can articulate with respect to the emitter 106 and the jaws at the distal end of the surgical device 102. The articulated configuration may include, for example, a multi-articulated vertebral-like structure. In some embodiments, a three-dimensional camera can be used to triangulate one or more distances to the surface 105.

[0050] In Figure 1, d wis the camera-to-critical-structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the critical structure 101, d A is the depth of the critical structure 101 below the surface 105 of the tissue 103 (e.g., the distance between a portion of the surface 105 closest to the surgical device 102 and the critical structure 101). The flight time of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 is the camera-to-critical-structure distance d w is configured to determine.

[0051] As shown in FIG. 2, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 A is the distance d w , as well as the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the known distance d x ) is determined by triangulation, and the distance d e and d A which is the sum of the distances d y can be determined. Additionally or alternatively, the flight time from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or waveform range) can be utilized to determine the camera-to-critical-structure distance d w , and a second waveform (or waveform range) can be utilized to determine the distance to the surface 105 of the tissue 103. In such an example, different waveforms can be utilized to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0052] Additionally or alternatively, the distance d A can be determined from ultrasound, registered magnetic resonance imaging (MRI), or computerized tomography (CT) scans. In still other examples, the distance d ASince the detection signal received by the imaging device 120 may vary based on the type of material, for example, the type of tissue 103, it can be determined by spectral imaging. For example, fat can reduce the detection signal by a first method or by a first amount, and collagen can reduce the detection signal by a different second method or by a second amount.

[0053] In another embodiment of the surgical visualization system 160 illustrated in Figure 3, the surgical device 162, rather than the imaging device 120, includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform emitter 123 is located at a distance d from a common device such as the surgical device 162, as further described herein. t , and d w It is configured to emit a waveform for determining the distance d from the surface 105 of tissue 103 to the surface of critical structure 101. A This can be determined as follows: d A =d w -d t

[0054] The surgical visualization system 100 includes a control system configured to control various aspects of the surgical visualization system 100. Figure 4 illustrates one embodiment of a control system 133 that can be used as a control system for the surgical visualization system 100 (or other surgical visualization systems described herein). The control system 133 includes a control circuit 132 configured to signal-communicate with a memory 134. The memory 134 is configured to store instructions that can be executed by the control circuit 132, such as instructions for determining and / or recognizing critical structures (e.g., critical structure 101 in Figure 1), instructions for determining and / or calculating one or more distances and / or three-dimensional digital representations, and instructions for communicating information to a physician. Thus, the instructions stored in the memory 134 constitute a computer program product that, when executed by the processor, includes instructions that cause the processor to execute as described above. Such instructions may also be stored on any computer-readable medium (such as an optical disc, SD card, USB drive, or memory of a separate device), copied from there to the memory 134, or executed directly. The copy or direct execution process involves creating a data carrier signal that carries the computer program product. As in this illustrated embodiment, memory 134 can store surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141, but memory 134 can store any combination of logic 136, 138, 140, and 141, and / or various logics can be combined together. Control system 133 also includes imaging system 142 (e.g., imaging system including imaging device 120 in Figure 1) including camera 144, display 146 (e.g., monitor, computer tablet screen), and control unit 148 for camera 144 and display 146. Camera 144 includes image sensor 135 (e.g., waveform sensor 122) (e.g., visible light, spectral imager, three-dimensional lens, etc.) configured to receive signals from various light sources emitting light in various visible and invisible spectra.The display 146 is configured to show the physician real, virtual, and / or virtually augmented images and / or information.

[0055] In exemplary embodiments, the image sensor 135 is a solid-state electronic device containing up to several million individual photodetector segments called pixels. The technology of the image sensor 135 is classified into one of two categories: charge-coupled device (CCD) imagers and complementary metal oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being a more recent development in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (marketed as "sCOMS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. The CCD and CMOS image sensors 135 are sensitive to wavelengths in the range of approximately 350 nm to 1050 nm, such as in the range of approximately 400 nm to 1000 nm. Those skilled in the art will recognize that while a value is not precisely that value, it may still be considered approximately that value due to various reasons such as the sensitivity of the measuring instrument and manufacturing tolerances. CMOS sensors are generally more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Therefore, there are two types of color CCD cameras: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to separate incident light into a series of colors and employing interpolation algorithms to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing prisms, directing each section of the incident spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a distinct RGB intensity value rather than using an algorithm to determine the color. Three-chip cameras offer very high resolution.

[0056] The control system 133 also includes emitters (e.g., emitter 106) each containing a spectral light source 150 and a structured light source 152, each operably coupled to the control circuit 133. A single light source can be pulsed to emit wavelengths of light within the spectral light source 150 range and wavelengths of light within the structured light source 152 range. Alternatively, a single light source can be pulsed to supply wavelengths of light in the invisible spectrum (e.g., infrared spectral light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. The tissue identification logic 140 is configured to identify key structures (e.g., key structures 101 in Figure 1) via data from the spectral light source 150 received by the image sensor 135 of the camera 144. The surface mapping logic 136 is configured to determine the surface contour of a visible tissue (e.g., tissue 103) based on reflected structured light. By measuring time of flight, the distance determination logic 141 is configured to determine the distance to one or more visible tissues and / or critical structures. The outputs from each of the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 are provided to the imaging logic 138, which is configured to combine, integrate, and / or superimpose them so that they are communicated to the physician via the display 146 of the imaging system 142.

[0057] The control circuit 132 can have various configurations. Figure 5 illustrates one embodiment of a control circuit 170 that can be used as a control circuit 132 configured to control aspects of the surgical visualization system 100. The control circuit 170 is configured to implement various processes described herein. The control circuit 170 includes a microcontroller, which includes a processor 172 (e.g., a microprocessor or microcontroller) operably coupled to a memory 174. The memory circuit 174 is configured to store machine-executable instructions, which, when executed by the processor 172, cause the processor 172 to execute machine instructions for implementing various processes described herein. The processor 172 can be any one of several single-core or multi-core processors known in the art. The memory circuit 174 can include volatile and non-volatile storage media. The processor 172 includes an instruction processing unit 176 and an arithmetic unit 178. The instruction processing unit 176 is configured to receive instructions from the memory circuit 174.

[0058] The surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141 can have various configurations. Figure 6 illustrates one embodiment of a combinational logic circuit 180 configured to control aspects of the surgical visualization system 100 using logic such as one or more of the surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141. The combinational logic circuit 180 includes a finite state machine including a combinational logic 182 configured to receive data associated with surgical devices (e.g., surgical device 102 and / or imaging device 120) at input 184, process the data by the combinational logic 182, and provide the output 184 to a control circuit (e.g., control circuit 132).

[0059] Figure 7 illustrates one embodiment of a sequential logic circuit 190 configured to control aspects of a surgical visualization system 100 using logic such as one or more of the following: surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141. The sequential logic circuit 190 includes a finite state machine comprising combinational logic 192, memory 194, and a clock 196. The memory 194 is configured to store the current state of the finite state machine. The sequential logic circuit 190 can be synchronous or asynchronous. The combinational logic 192 is configured to receive data associated with surgical devices (e.g., surgical device 102 and / or imaging device 120) at input 426, process the data by the combinational logic 192, and provide an output 499 to a control circuit (e.g., control circuit 132). In some embodiments, the sequential logic circuit 190 may include a combination of a processor (e.g., processor 172 in Figure 5) and a finite state machine to perform various processes described herein. In some embodiments, a finite state machine may include a combination of combinational logic circuits (e.g., combinational logic circuit 192 in Figure 7) and sequential logic circuits 190.

[0060] Figure 8 illustrates another embodiment of the surgical visualization system 200. The surgical visualization system 200 is generally configured and used similarly to the surgical visualization system 100 of Figure 1, and includes, for example, a surgical device 202 and an imaging device 220. The imaging device 220 includes, for example, a spectral light emitter 223 configured to emit spectral light of multiple wavelengths to acquire spectral images of hidden structures. The imaging device 220 may also include a three-dimensional camera and associated electronic processing circuits. The surgical visualization system 200 is shown being used in surgery to identify certain critical structures, such as the ureter 201a which is invisible on the surface 205 of organ 203, and blood vessels 201b within organ 203 (in this example, the uterus), and to facilitate their avoidance.

[0061] The surgical visualization system 200 uses structured light to measure the emitter-tissue distance d from the emitter 206 on the surgical device 202 to the surface 205 of the uterus 203. e The surgical visualization system 200 is configured to determine the emitter-tissue distance d e Based on this, the device-tissue distance d from the surgical device 202 to the surface 205 of the uterus 203 t It is configured to estimate the tissue-ureteral distance d from the ureter 201a to the surface 205. A , and the camera-ureteral distance d from the imaging device 220 to the ureter 201a w It is configured to determine the distance d. As described herein, for example, with respect to the surgical visualization system 100 of Figure 1, the surgical visualization system 200 uses, for example, spectral imaging and time-of-flight sensors to determine the distance d w It is configured to determine the tissue-ureteral distance d based on other distances and / or surface mapping logic described herein. In various embodiments, the surgical visualization system 200 determines the tissue-ureteral distance d based on other distances and / or surface mapping logic described herein. A (That is, depth) can be determined (for example, by triangulation).

[0062] As described above, the surgical visualization system includes a control system configured to control various aspects of the surgical visualization system. The control system can have various configurations. Figure 9 illustrates one embodiment of a control system 600 for a surgical visualization system, such as the surgical visualization system 100 of Figure 1, the surgical visualization system 200 of Figure 8, or other surgical visualization systems described herein. The control system 600 is a transformation system that integrates spectral signature tissue identification and structured phototissue localization to identify the structure(s) when important structures are obscured by tissues, e.g., fat, connective tissue, blood tissue, and / or organs(s), and / or blood, and / or to detect tissue variability, such as distinguishing tumors and / or non-healthy tissue from healthy tissue within organs.

[0063] The control system 600 is configured to implement a hyperspectral imaging visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field. The control system 600 includes a conversion logic circuit 648 configured to convert tissue data into information usable by surgeons and / or other healthcare professionals. For example, key structures within an anatomical structure can be identified using wavelength-based variable reflectivity to obscuring materials. Furthermore, the control system 600 is configured to combine identified spectral signatures with structural optical data within an image. For example, the control system 600 can be employed to create a three-dimensional dataset for surgical applications in a system using augmented image overlays. The technology can be employed both during and before surgery, incorporating additional visual information. In various embodiments, the control system 600 is configured to provide alerts to the physician when one or more key structures are in proximity. Various algorithms can be employed to guide robotically automated and semi-automated approaches based on the surgical procedure and proximity to key structures.

[0064] The projected light array is employed by the control system 600 to determine the shape and movement of the tissue during surgery. Alternatively, flash lidar may be used for tissue surface mapping.

[0065] As described above, the control system 600 is configured to detect critical structures, which may include one or more critical structures, provide an image overlay of the critical structures, and measure the distance to the surface of the visible tissue and the distance to embedded / buried critical structures. The control system 600 can measure the distance to the surface of the visible tissue, or detect critical structures and provide an image overlay of the critical structures.

[0066] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 may be a field programmable gate array (FPGA) or another preferred circuit configuration, such as the configuration described with respect to Figures 6, 7, and 8. The spectral control circuit 602 includes a processor 604 configured to receive a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, and for example, C / C++ code may be used. The video input processor 606 is configured to receive video in control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608, which includes, for example, a hyperspectral video out of interface control (metadata) data. The video output processor 608 is configured to provide the video output signal to an image overlay controller 610.

[0067] The video input processor 606 is operably coupled to the patient-side camera 612 via a patient isolation circuit 614. The camera 612 includes a solid-state image sensor 634. The patient isolation circuit 614 may include a number of transformers to isolate the patient from other circuits in the system. The camera 612 is configured to receive intraoperative images via an optical element 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor or another image sensor technology such as those discussed herein in relation to Figure 4. The camera 612 is configured to output 613 images with a 14-bit / pixel signal. Those skilled in the art will recognize that higher or lower resolutions may be employed. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which in this illustrated embodiment employs a camera hardware register 618 and a Nios2 coprocessor 620 configured to process the camera output signal 613. The video input processor 606 and the laser pulse control circuit 622 are provided with a color RGB fused output signal.

[0068] The laser pulse control circuit 622 is configured to control the laser light engine 624. The laser light engine 624 is configured to output light at multiple wavelengths (λ1, λ2, λ3...λn), including near-infrared (NIR). The laser light engine 624 can operate in multiple modes. For example, the laser light engine 624 can operate in two modes. In a first mode, for example, normal operation mode, the laser light engine 624 is configured to output an illumination signal. In a second mode, for example, specific mode, the laser light engine 624 is configured to output RGBG light and NIR light. In various embodiments, the laser light engine 624 can operate in polarization mode.

[0069] The optical output 626 from the laser light engine 624 is configured to illuminate a targeted anatomical structure within the surgical site 627 during surgery. The laser pulse control circuit 622 is also configured to control a laser pulse controller 628 for a laser pattern projector 630 configured to project a laser light pattern 631, such as a grid or pattern of lines and / or dots, at a predetermined wavelength (λ2) onto surgical tissue or organs in the surgical site 627. The camera 612 is configured to receive the patterned and reflected light output through the camera optical element 632. The image sensor 634 is configured to convert the received light into a digital signal.

[0070] The color RGB fusion circuit 616 is also configured to output signals to the image overlay controller 610 and to a video input module 636 for reading the laser beam pattern 631 projected by the laser pattern projector 630 onto the targeted anatomical structure at the surgical site 627. The processing module 638 is configured to process the laser beam pattern 631 and output a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 is also configured to output a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the targeted anatomical structure at the surgical site.

[0071] The first video output signal 640 and the second video output signal 642 include data representing the location of important structures on a three-dimensional surface model, provided to the integrated module 643. Combined with the data from the video output processor 608 of the spectral control circuit 602, the integrated module 643 calculates the distance to the buried important structures (e.g., distance d in Figure 1). AThe system is configured to determine (for example, by a triangulation algorithm 644) the distance to the buried critical structure, which can be provided to the image overlay controller 610 via the video output processor 646. The aforementioned transformation logic may include a transformation logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0072] Preoperative data 650 obtained from CT or MRI scans can be used to align or position specific three-dimensionally deformable tissues in various examples. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610, so that this information can be superimposed on the view from the camera 612 and provided to the video monitor 652. Embodiments of preoperative data alignment are further described in U.S. Patent Application Publication No. 2020 / 0015907, “Integration of Imaging Data,” filed September 11, 2018, which is incorporated herein by reference in its entirety.

[0073] The video monitor 652 is configured to output an integrated / enlarged view from the image overlay controller 610. The physician can select and / or switch between different views on one or more displays. In an illustrated embodiment, on a first display 652a, which is a monitor, the physician can switch between (A) a view showing a three-dimensional rendering of visible tissue and (B) an enlarged view in which one or more hidden critical structures are depicted on top of the three-dimensional rendering of visible tissue. In an illustrated embodiment, on a second display 652b, which is a monitor, the physician can switch between, for example, one or more hidden critical structures and / or distance measurements to the surface of visible tissue.

[0074] Various surgical visualization systems described herein can be used to visualize various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be obscured by EMR in the visible portion of the spectrum. The surgical visualization system can utilize a spectral imaging system, as described above, which can be configured to visualize different types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the tissue's absorption coefficients across various EMR wavelengths. The spectral imaging system can be configured to characterize the tissue type of the visualized tissue based on a specific combination of constituent materials.

[0075] Figure 10 shows Graph 300 illustrating how the absorption coefficients of various biomaterials vary across the EMR wavelength spectrum. In Graph 300, the vertical axis 302 represents the absorption coefficient of the biomaterial in cm². -1 The units are expressed, and the horizontal axis 304 represents the EMR wavelength in μm. In this graph 300, the first line 306 represents the absorption coefficient of water at various EMR wavelengths, the second line 308 represents the absorption coefficient of protein at various EMR wavelengths, the third line 310 represents the absorption coefficient of melanin at various EMR wavelengths, the fourth line 312 represents the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, the fifth line 314 represents the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and the sixth line 316 represents the absorption coefficient of collagen at various EMR wavelengths. Since different tissue types have different combinations of constituent materials, the tissue types visualized by the surgical visualization system can be identified and differentiated according to specific combinations of constituent materials detected. Therefore, the spectral imaging system of a surgical visualization system can be configured to emit EMR at several different wavelengths, determine the constituent materials of the tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on a specific detected combination of constituent materials.

[0076] Figure 11 illustrates one embodiment of the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures. In Figure 11, a spectral emitter 320 (e.g., spectral light source 150 in Figure 4) is used by the imaging system to visualize the surgical site 322. EMR emitted by the spectral emitter 320 and reflected from the tissues and / or structures of the surgical site 322 is received by an image sensor (e.g., image sensor 135 in Figure 4) to visualize the tissues and / or structures, which may be either visible (e.g., located on the surface of the surgical site 322) or obscured (e.g., beneath other tissues and / or structures in the surgical site 322). In this embodiment, the imaging system (e.g., imaging system 142 in Figure 4) visualizes tumors 324, arteries 326, and various abnormalities 328 (e.g., tissues that cannot be identified against known or expected spectral signatures) for each of different tissue / structure types, based on spectral signatures characterized by different absorbent properties (e.g., absorption coefficients) of the constituent materials. The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system (e.g., display 146 of imaging system 142 in Figure 4), on a main display (e.g., main display 819 in Figure 19), on a non-sterile display (e.g., non-sterile displays 807, 809 in Figure 19), on a surgical hub display (e.g., display of surgical hub 806 in Figure 19), on a device / instrument display, and / or on another display.

[0077] The imaging system can be configured to adjust or update the visualization of the displayed surgical site according to the identified tissue and / or structural type. For example, as shown in Figure 11, the imaging system can display a margin 330 associated with a tumor 324 visualized on a display screen associated with or coupled to the imaging system, on the main display, on a non-sterile display, on the surgical hub display, on the device / instrument display, and / or on another display. The margin 330 can indicate the area or amount of tissue to be excised to ensure complete removal of the tumor 324. The control system of the surgical visualization system (e.g., control system 133 in Figure 4) can be configured to control or update the dimensions of the margin 330 based on the tissue and / or structure identified by the imaging system. In the illustrated embodiment, the imaging system identifies several anomalies 328 within the field of view (FOV). The control system can then adjust the displayed margin 330 to a first updated margin 332 having sufficient dimensions to encompass these anomalies 328. Furthermore, the imaging system identifies artery 326 that partially overlaps with the initially displayed margin 330 (as indicated by the highlighted region 334 of artery 326). Therefore, the control system can adjust the displayed margin to a second updated margin 336 that has sufficient dimensions to encompass the relevant portion of artery 326.

[0078] Tissues and / or structures can also be imaged or characterized according to these reflectivity characteristics across the EMR wavelength spectrum, in addition to or instead of the absorptivity characteristics described above with respect to Figures 10 and 11. For example, Figures 12, 13, and 14 illustrate various graphs of the reflectivity of different types of tissues or structures across various EMR wavelengths. Figure 12 is a graph representation of an exemplary ureteral signature against an obscuring object. Figure 13 is a graph representation of an exemplary arterial signature against an obscuring object. Figure 14 is a graph representation of an exemplary nerve signature against an obscuring object. The plots in Figures 12, 13, and 14 show the reflectivity of specific structures (ureters, arteries, and nerves) as a function of wavelength (nm) against the corresponding reflectivity of fat, lung tissue, and blood at the corresponding wavelengths. These graphs are for illustrative purposes only, and it should be understood that other organizations and / or structures may include corresponding detectable reflectance signatures that enable identification and visualization of the organization and / or structure.

[0079] Selective wavelengths for spectral imaging can be identified and utilized based on anticipated important structures and / or occluding elements at the surgical site (e.g., "selective spectral" imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized so that information can be acquired in real time and used during surgery. Wavelengths can be selected by the physician or by the control circuit based on user input, e.g., from the physician. In certain examples, wavelengths can be selected based on machine learning and / or big data accessible to the control circuit, e.g., via the cloud or surgical hub.

[0080] Figure 15 illustrates one embodiment of spectral imaging of tissue used in surgery to measure the distance between a waveform emitter and a critical structure obscured by tissue. Figure 15 shows one embodiment of a time-of-flight sensor system 404 utilizing waveforms 424, 425. The time-of-flight sensor system 404 can be incorporated into a surgical visualization system, for example, as sensor system 104 of the surgical visualization system 100 in Figure 1. The time-of-flight sensor system 404 includes a waveform emitter 406 and a waveform receiver 408 on the same surgical device 102 (e.g., emitter 106 and receiver 108 on the same surgical device 402 in Figure 1). The emitted wave 400 extends from the emitter 406 to the critical structure 401 (e.g., critical structure 101 in Figure 1), and the received wave 425 is reflected back from the critical structure 401 to the receiver 408. In this illustrated embodiment, the surgical device 402 is positioned through a trocar 410 that extends into the patient's cavity 407. In this illustrated embodiment, a trocar 410 is used, but other trocars or other access devices may be used, or no access device may be used at all.

[0081] Waveforms 424 and 425 are configured to penetrate the occluding tissue 403, for example, by having wavelengths within the NIR or SWIR spectrum. A spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal is emitted from the emitter 406, as indicated by a first arrow 407 pointing distally, and can penetrate the tissue 403 in which the critical structure 401 is occluded. The emitted waveform 424 is reflected by the critical structure 401, as indicated by a second arrow 409 pointing proximal. The received waveform 425 may be delayed due to the distance d between the distal end of the surgical device 402 and the critical structure 401. Waveforms 424 and 425 can be selected to target the critical structure 401 in the tissue 403 based on the spectral signature of the critical structure 401, as described herein. The emitter 406 is configured to provide a binary signal representing on and off, as shown in Figure 16, for example, and can be measured by the receiver 408.

[0082] Based on the delay between the emitted wave 424 and the received wave 425, the time-of-flight sensor system 404 is configured to determine the distance d. The time-of-flight timing diagram 430 of the emitter 406 and receiver 408 in Figure 15 is shown in Figure 16. The delay is a function of the distance d, which is given as follows:

[0083]

number

[0084] The time of flight of waveforms 424 and 425 corresponds to distance d in Figure 15. In various examples, an additional emitter / receiver and / or pulse signal from emitter 406 can be configured to emit an opaque signal. The opaque signal can be configured to determine the distance from emitter 406 to the surface 405 of the covering tissue 403. In various examples, the depth of critical structure 401 is d A =d w -d t This can be determined by d A =This is the depth of important structure 401, d w = This is the distance from emitter 406 to critical structure 401 (d in Figure 15), and d t = This is the distance from the emitter 406 (on the distal end of the surgical device 402) to the surface 405 of the covering tissue 403.

[0085] Figure 17 illustrates another embodiment of a time-of-flight sensor system 504 utilizing waves 524a, 524b, 524c, 525a, 525b, and 525c. The time-of-flight sensor system 504 can be incorporated into a surgical visualization system, for example, as sensor system 104 in the surgical visualization system 100 of Figure 1. The time-of-flight sensor system 504 includes a waveform emitter 506 and a waveform receiver 508 (e.g., emitter 106 and receiver 108 in Figure 1). The waveform emitter 506 is positioned on a first surgical device 502a (e.g., surgical device 102 in Figure 1), and the waveform receiver 508 is positioned on a second surgical device 502b. The surgical devices 502a and 502b are positioned through a first trocar 510a and a second trocar 510b, respectively, extending into the patient's cavity 507. While trocars 510a and 510b are used in this illustrated embodiment, other trocars or other access devices may be used, or no access device may be used at all. The emitted waves 524a, 524b, and 524c extend from the emitter 506 toward the surgical site, and the received waves 525a, 525b, and 525c are reflected from the receiver 508 by various structures and / or surfaces at the surgical site.

[0086] Different emitted waves 524a, 524b, and 524c are configured to target different types of material at the surgical site. For example, wave 524a targets the covering tissue 503, wave 524b targets a first critical structure 501a (e.g., critical structure 101 in Figure 1), which is a blood vessel in this illustrated embodiment, and wave 524c targets a second critical structure 501b (e.g., critical structure 101 in Figure 1), which is a cancerous tumor in this illustrated embodiment. The wavelengths of waves 524a, 524b, and 524c can be wavelengths of visible light, NIR, or SWIR spectra. For example, visible light can be reflected by the surface 505 of the tissue 503, and NIR and / or SWIR waveforms can be transmitted through the surface 505 of the tissue 503. In various embodiments, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from the emitter 506 as described herein. Waves 524b and 524c can be selected to target key structures 501a and 501b within tissue 503 based on their spectral signatures, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, which are incorporated herein by reference.

[0087] The emitted waves 524a, 524b, and 524c are reflected from the target material, namely the surface 505, the first critical structure 501a, and the second structure 501b, respectively. The received waveforms 525a, 525b, and 525c are reflected at a distance d 1a d 2a d 3a d 1b d 2b d 2c There is a possibility of delays due to this.

[0088] In a time-of-flight sensor system 504 in which the emitter 506 and receiver 508 can be positioned individually (for example, on separate surgical devices 502a, 502b and / or controlled by a separate robotic arm), various distances d 1a d 2a d3a d 1b d 2b d 2c This can be calculated from the known positions of the emitter 506 and receiver 508. For example, when surgical devices 502a and 502b are robotically controlled, their positions can be known. With knowledge of the positions of the emitter 506 and receiver 508, as well as the time of the photon stream targeting a specific tissue and the information of that specific response received by receiver 508, the distance d can be calculated. 1a d 2a d 3a d 1b d 2b d 2c This makes it possible to determine the distance. In one embodiment, the distance to the obscured critical structures 501a, 501b can be determined by triangulation using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 504 can determine various distances.

[0089] The view provided to the physician on a display or other means allows the receiver 508 to be rotated, for example, in a plane perpendicular to the axis of a selected target structure 503, 501a, or 501b, so that the center of mass of the target structure in the resulting image remains constant. Such orientation allows for the rapid communication of one or more associated distances and / or viewpoints with respect to the target structure. For example, as shown in Figure 17, the surgical site is displayed from a viewpoint where the critical structure 501a is perpendicular to the view plane (e.g., blood vessels are facing inward and outward from the page). Such orientation can be the default setting. However, the view may be rotated or otherwise adjusted by the physician. In certain examples, the physician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0090] As in this illustrated embodiment, the receiver 508 can be mounted on a trocar 510b (or other access device) through which the surgical device 502b is positioned. In other embodiments, the receiver 508 can be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 508 can be mounted on a movable arm separate from the robotic surgical system controlling the surgical device 502a, or on an operating room (OR) table or fixture that can be intraoperatively aligned to the robotic coordinate plane. In such examples, the positions of the emitter 506 and the receiver 508 can be recorded on the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 504.

[0091] The combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS), known as TOF-NIRS, which enables the measurement of time-resolved profiles of NIR light with nanosecond resolution, can be found in *Time-Of-Flight Near-Infrared Spectroscopy For Nondestructive Measurement Of Internal Quality In Grapefruit*, *Journal of the American Society for Horticultural Science*, May 2013, vol. 138, no. 3, pp. 225-228, which is incorporated herein by reference in its entirety.

[0092] The visualization system, its forms, and embodiments of use are described in U.S. Patent Application Publication No. 2020 / 0015923, “Surgical Visualization Platform,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015900, “Controlling An Emitter Assembly Pulse Sequence,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015668, “Singular EMR Source Emitter Assembly,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015925, “Combination Emitter And Camera Assembly,” filed on September 11, 2018; and “Surgical Visualization With Proximity Tracking U.S. Patent Application Publication No. 2020 / 00015899, entitled "Features", U.S. Patent Application Publication No. 2020 / 00015903, entitled "Surgical Visualization Of Multiple Targets", filed on September 11, 2018, and U.S. Patent Application Publication No. 10,792, entitled "Visualization Of Surgical Devices", filed on September 11, 2018.U.S. Patent Application Publication No. 034, entitled "Operative Communication Of Light" (2020 / 0015897), filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015924, entitled "Robotic Light Projection Tools", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015898, entitled "Surgical Visualization Feedback System", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015906, entitled "Surgical Visualization And Monitoring", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015907, entitled "Integration Of Imaging Data", filed on September 11, 2018; "Robotically-Assisted Surgical U.S. Patent No. 10,925,598, titled "Surgical Surgical Systems"; U.S. Patent Publication No. 2020 / 0015901, filed September 11, 2018, titled "Safety Logic For Surgical Surgical Systems"; U.S. Patent Publication No. 2020 / 0015914, filed September 11, 2018, titled "Robotic Systems With Separate Photoacoustic Receivers"; U.S. Patent Publication No. 2020 / 0015902, filed September 11, 2018, titled "Force Sensor Through Structured Light Deflection"; U.S. Patent Publication No. 2019 / 0201136, filed December 4, 2018, titled "Method Of Hub Communication"; U.S. Patent Publication No. 2019 / 0201136, filed December 30, 2019, titled "Analyzing Surgical Trends By A Surgical U.S. Patent Application No. 16 / 729,772, titled "System", and U.S. Patent Application No. 16 / 729, filed on December 30, 2019, titled "Dynamic Surgical Visualization Systems",U.S. Patent Application No. 16 / 729,744, filed December 30, 2019, entitled "Visualization Systems Using Structured Light"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue"; U.S. Patent Application No. 16 / 729,729, filed December 30, 2019, entitled "Surgical Systems For Proposing And Corroborating Organ Portion Removals"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical Systems For Generating Three U.S. Patent Application No. 16 / 729,751 entitled "Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto", U.S. Patent Application No. 16 / 729,740 entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data", filed on December 30, 2019, and U.S. Patent Application No. 16 / 729,740 entitled "Adaptive Surgical System Control According To Surgical Smoke Cloud Characteristics", filed on December 30, 2019.U.S. Patent Application No. 16 / 729,796, filed December 30, 2019, entitled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics"; U.S. Patent Application No. 16 / 729,803, filed December 30, 2019, entitled "Adaptive Visualization By A Surgical System"; U.S. Patent Application No. 16 / 729,807, filed December 30, 2019, entitled "Method Of Using Imaging Devices In Surgery"; U.S. Provisional Patent Application No. 63 / 249,652, filed September 29, 2021, entitled "Surgical Devices, Systems, and Methods Using Fiducial Identification and Tracking"; U.S. Patent Application No. 737, filed December 30, 2019, entitled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics"; U.S. Patent Application No. 16 / 729,803, filed December 30, 2019, entitled "Adaptive Visualization By A Surgical System"; U.S. Patent Application No. 16 / 729,807, filed December 30, 2019, entitled "Method Of Using Imaging Devices In Surgery"; U.S. Provisional Patent Application No. 63 / 249,652, filed September 29, 2021, entitled "Surgical Devices, Systems, and Methods for Control of One Visualization with U.S. Provisional Patent Application No. 63 / 249,658, titled "Another", U.S. Provisional Patent Application No. 63 / 249,870, titled "Methods and Systems for Controlling Cooperative Surgical Instruments", filed on September 29, 2021, U.S. Provisional Patent Application No. 63 / 249,881, titled "Methods and Systems for Controlling Cooperative Surgical Instruments with Variable Surgical Site Access Trajectories", filed on September 29, 2021, U.S. Provisional Patent Application No. 63 / 249,877, titled "Methods and Systems for Controlling Cooperative Surgical Instruments", filed on September 29, 2021, U.S. Provisional Patent Application No. 63 / 249,858, titled "Another", filed on September 29, 2021Issue No. 980, which is incorporated herein by reference in its entirety.

[0093] Surgical Hub The various visualization or imaging systems described herein can be incorporated into a system including a surgical hub. Generally, a surgical hub can span multiple healthcare facilities and be a component of a comprehensive digital healthcare system configured to provide integrated, comprehensive, and improved healthcare to a vast number of patients. The comprehensive digital healthcare system includes a cloud-based healthcare analytics system configured to interconnect multiple surgical hubs located across many different healthcare facilities. A surgical hub is configured to interconnect one or more elements, such as one or more surgical instruments used to perform medical procedures on patients and / or one or more visualization systems used while performing medical procedures. The surgical hub provides diverse functionality to improve the outcomes of medical procedures. Data on patients and medical procedures generated by various surgical devices, visualization systems, and surgical hubs can be transmitted to the cloud-based healthcare analytics system. This data can then be aggregated with similar data collected from many other surgical hubs, visualization systems, and surgical instruments located in other healthcare facilities. Through the cloud-based analytics system that analyzes the collected data, various patterns and correlations can be found. As a result, improvements in the techniques used to generate data can be made, and these improvements can then be disseminated to various surgical hubs, visualization systems, and surgical instruments. The interconnectivity of all the components described above may lead to improvements in medical procedures and practices that might not be found if many of the components were not interconnected in this way.

[0094] Examples of surgical hubs configured to receive, analyze, and output data, and methods for using such surgical hubs, are described in U.S. Patent Application Publication No. 2019 / 0200844, “Method Of Hub Communication, Processing, Storage And Display,” filed December 4, 2018; U.S. Patent Application Publication No. 2019 / 0200981, “Method Of Compressing Tissue Within A Stapling Device And Simultaneously Displaying The Location Of The Tissue Within The Jaws,” filed December 4, 2018; U.S. Patent Application Publication No. 2019 / 0201046, “Method For Controlling Smart Energy Devices,” filed December 4, 2018; and U.S. Patent Application Publication No. 2019 / 0201114, “Adaptive Control Program Updates For Surgical Hubs,” filed March 29, 2018.This is further described in U.S. Patent Application Publication No. 2019 / 0201140, “Surgical Hub Situational Awareness,” filed March 29, 2018; U.S. Patent Application Publication No. 2019 / 0206004, “Interactive Surgical Systems With Condition Handling Of Devices And Data Capabilities,” filed March 29, 2018; U.S. Patent Application Publication No. 2019 / 0206555, “Cloud-based Medical Analytics For Customization And Recommendations To A User,” filed March 29, 2018; and U.S. Patent Application Publication No. 2019 / 0207857, filed November 6, 2018, which are incorporated herein by reference in their entirety.

[0095] Figure 18 shows one embodiment of a bidirectional surgical system 700 implemented on a computer, comprising one or more surgical systems 702 and a cloud-based system (e.g., a cloud 704 which may include a remote server 713 coupled to a storage device 705). Each surgical system 702 includes at least one surgical hub 706 that communicates with the cloud 704. In one embodiment, as illustrated in Figure 18, the surgical system 702 includes a visualization system 708, a robotic system 710, and intelligent (or "smart") surgical instruments 712, which are configured to communicate with each other and / or with the hub 706. The intelligent surgical instruments 712 may include an imaging device. The surgical system 702 may include M hubs 706, N visualization systems 708, O robotic systems 710, and P intelligent surgical instruments 712, where M, N, O, and P are integers of one or more, which may be equal to or not equal to one or more of each other. Various exemplary intelligent surgical instruments and robotic systems are described herein.

[0096] The data received by the surgical hub from the surgical visualization system can be used in any of the following ways. In an exemplary embodiment, the surgical hub can receive data from the surgical visualization system when used with the patient in a surgical setting, for example, when used in the operating room during a surgical procedure. The surgical hub can use the data received in any one or more ways, as discussed herein.

[0097] The surgical hub can be configured to analyze the received data in real time in conjunction with the use of the surgical visualization system and, based on the analysis of the received data, to adjust and control one or more of the surgical visualization system and / or one or more intelligent surgical instruments used with the patient. Such adjustments may include, for example, adjusting one or more operational control parameters of the intelligent surgical instrument, having one or more sensors of one or more intelligent surgical instruments take measurements to help understand the patient's current physiological state and / or the current operational state of the intelligent surgical instrument, and other adjustments. Controlling and adjusting the operation of intelligent surgical instruments will be discussed further below. Examples of operational control parameters for intelligent surgical instruments include motor speed, cutting element speed, time, duration, energy application level, and light emission. Examples of surgical hubs, as well as examples of controlling and regulating the operation of intelligent surgical instruments, are described in U.S. Patent Application No. 16 / 729,772, “Analyzing Surgical Trends By A Surgical System,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,747, “Dynamic Surgical Visualization Systems,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,744, “Visualization Systems Using Structured Light,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,778, “System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue,” filed December 30, 2019; and “Surgical Systems For Proposing And Corroborating Organ Portion U.S. Patent Application No. 16 / 729, entitled “Removals”,U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ"; U.S. Patent Application No. 16 / 729,751, filed December 30, 2019, entitled "Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto"; U.S. Patent Application No. 16 / 729,740, filed December 30, 2019, entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data"; and "Adaptive Surgical System Control According To Surgical Smoke Cloud". U.S. Patent Application No. 16 / 729,737, titled "Characteristics", U.S. Patent Application No. 16 / 729,796, filed December 30, 2019, titled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics", U.S. Patent Application No. 16 / 729,803, filed December 30, 2019, titled "Adaptive Visualization By A Surgical System", and U.S. Patent Application No. 16 / 729, filed December 30, 2019, titled "Method Of Using Imaging Devices In Surgery".This is described in Patent No. 807. U.S. Patent Application No. 17 / 068,857, “Adaptive Responses From Smart Packaging Of Drug Delivery Absorbable Adjuncts,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,858, “Drug Administration Devices That Communicate With Surgical Hubs,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,859, “Controlling Operation Of Drug Administration Devices Using Surgical Hubs,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,863, “Patient Monitoring Using Drug Administration Devices,” filed October 13, 2020; and “Monitoring And Communicating Information Using Drug Administration Further details are provided in U.S. Patent Application No. 17 / 068,865, entitled “Devices,” and U.S. Patent Application No. 17 / 068,867, entitled “Aggregating And Analyzing Drug Administration Data,” filed on 13 October 2020, which are incorporated herein by reference in their entirety.

[0098] The surgical hub can be configured to provide visualization of received data on a display within the surgical setting, so that the physician in the surgical setting can view the data and thereby gain an understanding of the operation of the imaging devices used in the surgical setting. Such information provided through visualization may include text and / or images.

[0099] Figure 19 shows one embodiment of a surgical system 802, which includes a surgical hub 806 (e.g., surgical hub 706 in Figure 18 or other surgical hubs described herein), a robotic surgical system 810 (e.g., robotic surgical system 110 in Figure 1 or other robotic surgical systems described herein), and a visualization system 808 (e.g., visualization system 100 in Figure 1 or other visualization systems described herein). The surgical hub 806 can communicate with a cloud, as discussed herein. Figure 19 shows the surgical system 802 used to perform a surgical procedure on a patient lying on an operating table 814 in an operating room 816. The robotic system 810 includes a surgeon's console 818, a patient-side cart 820 (surgical robot), and a surgical hub 822 of the robotic system. The surgical hub 822 of the robotic system is generally configured similarly to the surgical hub 822 and can communicate with a cloud. In some embodiments, the surgical hub 822 and surgical hub 806 of the robotic system can be combined. The patient-side cart 820 allows a physician, such as a surgeon, nurse, and / or other physician, to operate an intelligent surgical tool 812 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 818. Images of the surgical site can be acquired by an imaging device 824 (e.g., imaging device 120 in Figure 1, or other imaging devices described herein), which can be operated by the patient-side cart 820 to change the orientation of the imaging device 824. Images of the surgical site can be processed using the robotic system's surgical hub 822 and then displayed to the surgeon through the surgeon's console 818.

[0100] The main display 819 is positioned in the sterile field of the operating room 816 and is configured to be visible to the operator at the operating table 814. In addition, in this illustrated embodiment, a visualization tower 818 may be positioned outside the sterile field. The visualization tower 818 includes a first non-sterile display 807 and a second non-sterile display 809, facing opposite directions from each other. The visualization system 808, guided by the surgical hub 806, is configured to utilize displays 807, 809, and 819 to coordinate the flow of information to physicians inside and outside the sterile field. For example, the surgical hub 806 can cause the visualization system 808 to display snapshots and / or videos of the surgical site, such as those acquired by the imaging device 824, on one or both of the non-sterile displays 807 and 809, while maintaining live video of the surgical site on the main display 819. The snapshots and / or videos on the non-sterile displays 807 and / or 809 can, for example, enable a non-sterile physician to perform diagnostic steps related to the surgical procedure.

[0101] The surgical hub 806 is configured to send diagnostic input or feedback entered by a non-sterile physician in the visualization tower 818 to the main display 819 in the sterile field, which can then be viewed by a sterile physician at the operating table 814. For example, the input can take the form of a modified snapshot and / or video displayed on the non-sterile display 807 or 809, which can then be sent to the main display 819 by the surgical hub 806.

[0102] The surgical hub 806 is configured to adapt the information flow to the display of the intelligent surgical instrument 812, as described in various U.S. patent applications incorporated herein by reference. Diagnostic inputs or feedback entered by a non-sterile operator in the visualization tower 818 can be sent by the surgical hub 806 to a display 819 in the sterile field, which can be viewed by the operator of the surgical instrument 812 and / or the physician in the sterile field.

[0103] The intelligent surgical instruments 812 and the imaging device 824, which is also an intelligent surgical tool, are used with the patient in surgical procedures as part of the surgical system 802. Other intelligent surgical instruments 812a that can be used in surgical procedures, for example, which are detachably coupled to the patient-side cart 820 and can communicate with the robotic surgical system 810 and the surgical hub 806, are also shown in Figure 19 as being available. Non-intelligent (or "lacking data processing capability") surgical instruments 817, such as scissors, trocars, cannulas, and scalpels that cannot communicate with the robotic surgical system 810 and the surgical hub 806, are also shown in Figure 19 as being available for use.

[0104] Operation of intelligent surgical instruments An intelligent surgical device may have algorithms stored on the intelligent surgical device, for example in its memory, configured to be executable, for example by its processor, on the intelligent surgical device's substrate, for example, to control the operation of the intelligent surgical device. In some embodiments, instead of, or in addition to, being stored on the intelligent surgical device, the algorithms may be stored on a surgical hub, for example in its memory, which is configured to communicate with the intelligent surgical device.

[0105] The algorithm is stored in the form of one or more sets of data points that define and / or represent commands, notifications, signals, etc., to control the functions of the intelligent surgical device. In some embodiments, data collected by the intelligent surgical device can be used by the intelligent surgical device, for example, by the processor of the intelligent surgical device, to modify at least one variable parameter of the algorithm. As discussed above, a surgical hub can communicate with the intelligent surgical device, and therefore data collected by the intelligent surgical device can be communicated to the surgical hub, and / or data collected by another device communicating with the surgical hub can be communicated to the surgical hub, and data can be communicated from the surgical hub to the intelligent surgical device. Thus, instead of, or in addition to, the intelligent surgical device being configured to modify stored variable parameters, the surgical hub can be configured to communicate at least one modified variable to the intelligent surgical device, either alone or as part of an algorithm, and / or the surgical hub can communicate commands to the intelligent surgical device and modify at least one variable as determined by the surgical hub.

[0106] At least one variable parameter lies between the data points of the algorithm, for example, included in the instructions for operating the intelligent surgical device, and each can therefore be modified by changing one or more of the stored data points of the algorithm. After at least one variable parameter is changed, subsequent execution of the algorithm follows the modified algorithm. Thus, the operation of the intelligent surgical device over time can be managed for the patient in order to increase the use of beneficial outcomes of the intelligent surgical device by considering the actual situation of the patient and the actual conditions and / or outcomes of the surgical procedure in which the intelligent surgical device is used. Changing at least one variable parameter is automated to improve patient outcomes. Thus, the intelligent surgical device can be configured to provide personalized medicine based on the patient and the patient's surrounding conditions in order to provide a smart system. In a surgical setting in which the intelligent surgical device is used during a surgical procedure, the automatic modification of at least one variable parameter can enable the intelligent surgical device to be controlled based on data collected during the surgical procedure, which can help ensure that the intelligent surgical device is used efficiently and accurately and / or help reduce the possibility of harming the patient by damaging important anatomical structures.

[0107] At least one variable parameter can be one of a variety of different operating parameters. Examples of variable parameters include motor speed, motor torque, energy level, energy application duration, tissue compression rate, jaw closure rate, cutting element speed, and load threshold.

[0108] Figure 20 illustrates one embodiment of an intelligent surgical instrument 900, which includes a memory 902 storing an algorithm 904 having at least one variable parameter. The algorithm 904 may be a single algorithm, or it may include multiple algorithms, for example, separate algorithms for different modes of operation of the surgical instrument, each algorithm having at least one variable parameter. The intelligent surgical instrument 900 may be the surgical device 102 in Figure 1, the imaging device 120 in Figure 1, the surgical device 202 in Figure 8, the imaging device 220 in Figure 8, the surgical device 402 in Figure 15, the surgical device 502a in Figure 17, the surgical device 502b in Figure 17, the surgical device 712 in Figure 18, the surgical device 812 in Figure 19, the imaging device 824 in Figure 19, or other intelligent surgical instruments. The surgical instrument 900 also includes a processor 906 configured to execute the algorithm 904 and control the operation of at least one mode of the surgical instrument 900. To execute algorithm 904, processor 906 is configured to execute a program stored in memory 902 to access multiple data points of algorithm 904 in memory 902.

[0109] The surgical instrument 900 also includes a communication interface 908, such as a wireless transceiver or other wired or wireless communication interface, configured to communicate with another device, such as a surgical hub 910. The communication interface 908 can be configured to enable one-way communication, such as providing data to a remote server (e.g., a cloud server or other server) and / or a local surgical hub server, and / or receiving instructions or commands from the remote server and / or the local surgical hub server, or two-way communication, such as providing information, messages, data, etc., about the surgical instrument 900 and / or the data stored therein, and receiving instructions from a physician, etc. It can be configured to enable two-way communication, such as receiving instructions from a remote server regarding software updates, a local surgical hub server regarding software updates, etc.

[0110] The surgical instrument 900, simplified in Figure 20, may include additional components such as a bus system, a handle, an elongated shaft having an end effector at its distal end, and a power supply. The processor 906 may also be configured to execute instructions stored in memory 902 to control the device 900, including other electrical components such as a communication interface 908, an audio speaker, and a user interface.

[0111] The processor 906 is configured to modify at least one variable parameter of algorithm 904 so that subsequent executions of algorithm 904 conform to the modified at least one variable parameter. To modify at least one variable parameter of algorithm 904, the processor 906 is configured to modify or update the data point of at least one variable parameter in memory 902. The processor 906 can be configured to modify at least one variable parameter of algorithm 904 in real time in conjunction with the use of the surgical device 900 during a surgical procedure, which can address real-time requirements.

[0112] In addition to or instead of the processor 906 modifying at least one variable parameter, the processor 906 may be configured to modify algorithm 904 and / or at least one variable parameter of algorithm 904 in response to instructions received from the surgical hub 910. In some embodiments, the processor 906 is configured to modify at least one variable parameter only after communicating with and receiving instructions from the surgical hub 910, which may help ensure coordinated action of the surgical instrument 900 with other aspects of the surgical procedure in which the surgical instrument 900 is used.

[0113] In an exemplary embodiment, the processor 906 executes algorithm 904 to control the operation of the surgical instrument 900, modifies at least one variable parameter of algorithm 904 based on real-time data, and executes algorithm 904 to control the operation of the surgical instrument 900 after modifying at least one variable parameter.

[0114] Figure 21 illustrates one embodiment of a method 912 for using a surgical instrument 900, which includes changing at least one variable parameter of algorithm 904. A processor 906 controls the operation of the surgical instrument 900 by executing algorithm 904 stored in memory 902 (914). Based on either subsequently known data and / or subsequently collected data, the processor 904 changes at least one variable parameter of algorithm 904 as discussed above (916). After changing at least one variable parameter, the processor 906 controls the operation of the surgical instrument 900 by executing algorithm 904 using the changed at least one variable parameter (918). The processor 904 can change at least one variable parameter any number of times, for example, 0, 1, 2, 3 times 916 while performing a surgical procedure. In any part of method 912, the surgical instrument 900 may communicate with one or more computer systems, such as a surgical hub 910, a cloud server, or other remote servers, using the communication interface 908 to provide data thereto and / or receive instructions therefrom.

[0115] Situational awareness The behavior of intelligent surgical instruments can be altered based on the patient's situational awareness. The behavior of intelligent surgical instruments can also be manually altered by the user of the intelligent surgical instrument handling the instrument differently, providing different inputs to the instrument, or stopping the use of the instrument. Additionally or alternatively, the behavior of intelligent surgical instruments can be automatically altered by the instrument's algorithm, for example, by changing at least one variable parameter of the algorithm. As described above, the algorithm can be automatically adjusted without user input requesting the change. Automating adjustments during a surgical procedure can help save time, allow the physician to focus on other aspects of the surgical procedure, and / or simplify the process of using the surgical instrument for the physician, each of which can improve patient outcomes by avoiding critical structures, controlling the surgical instrument considering the types of tissue being used and / or nearby, etc.

[0116] The visualization systems described herein can be used as part of a context-aware system that may be embodied or performed by a surgical hub, e.g., surgical hub 706, surgical hub 806, or other surgical hubs described herein. In particular, characterizing, identifying, and / or visualizing surgical instruments (including their position, orientation, and actions), tissues, structures, users, and / or other things located in the surgical field or operating room can be used by the context-aware system to provide contextual data that allows inferences to be made about the type of surgical procedure or process being performed, the type of tissue and / or structure being manipulated by the surgeon or other physician, and other information. This contextual data can then be used by the context-aware system to provide alerts to the user, suggest subsequent processes or actions to be undertaken by the user, prepare surgical devices in anticipation of their use (e.g., activating an electrosurgical generator in anticipation of the use of an electrosurgical instrument in a subsequent process of the surgical procedure), and control the operation of intelligent surgical instruments (e.g., customizing the operating parameters of an algorithm for a surgical instrument, as will be discussed further below).

[0117] For example, by having at least one variable parameter in the modified algorithm, an intelligent surgical device that includes a control algorithm that responds to sensed data can be an improvement over a “data-dumb” device that operates without considering sensed data. However, some sensed data may be incomplete or indecisive when considered in isolation, without context, for example, the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the procedure context (e.g., the type of tissue being operated on or the type of procedure being performed), the algorithm may control the surgical device inaccurately or suboptimally when given sensed data without specific context. For example, the optimal form of a control algorithm for controlling a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different types of tissue have different properties (e.g., resistance to tearing, ease of cutting, etc.) and therefore respond differently to actions taken by the surgical instrument. Thus, even when the same measurement is sensed for a particular parameter, it may be desirable for the surgical instrument to take different actions. As one embodiment, the optimal mode of control for a surgical stapler in response to an unexpectedly high force being detected to close its end effector varies depending on whether the type of tissue is susceptible to tearing or resistant to tearing. For tear-prone tissues, such as lung tissue, the instrument's control algorithm optimally decelerates the motor in response to the unexpectedly high force required to close in order to avoid tearing the tissue, for example by changing a variable parameter that controls the motor speed or torque to slow down the motor. For tear-resistant tissues, such as stomach tissue, the instrument's control algorithm optimally accelerates the motor in response to the unexpectedly high force required to close in order to ensure that the end effector is properly clamped to the tissue, for example by changing a variable parameter that controls the motor speed or torque to speed up the motor.Without knowing whether lung or stomach tissue has been clamped, the algorithm may be modified to a suboptimal state or remain unchanged at all.

[0118] A surgical hub can be configured to derive information about the surgical procedure being performed based on data received from various data sources, and therefore to control modular devices accordingly. In other words, a surgical hub can be configured to infer information about the surgical procedure from received data, and then, based on the inferred context of the surgical procedure, to control modular devices operably coupled to the surgical hub. Modular devices can include any surgical device controllable by a context-aware system, such as a visualization system device (e.g., a camera, a display screen, etc.) or a smart surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, a surgical stapler, a smoke exhauster, etc.). Modular devices can include sensors configured to detect parameters associated with the patient on whom the device is being used, and / or parameters associated with the modular device itself.

[0119] Contextual information derived or inferred from received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon (or other physician), the type of tissue being operated on, or the body cavity being targeted by the surgical procedure. The surgical hub context-aware system can be configured to derive contextual information from data received from data sources in various different ways. In an exemplary embodiment, the contextual information received by the surgical hub context-aware system is associated with a specific control modulus, or set of control modulus, of one or more modular devices. Each control modulus corresponds to a variable parameter. In one embodiment, the context-aware system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from a database, patient monitoring devices, and / or modular devices) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another embodiment, the situation-aware system may include a lookup table that stores pre-characterized contextual information relating to a surgical procedure, associated with one or more inputs (or ranges of inputs) corresponding to that contextual information. In response to a query with one or more inputs, the lookup table may return the corresponding contextual information of the situation-aware system to control at least one modular device. In another embodiment, the situation-aware system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices when contextual information is provided as input.

[0120] A surgical hub incorporating a context-aware system can offer numerous advantages to the surgical system. One advantage is improved interpretation of perceived and collected data, which improves processing accuracy and / or data utilization during the surgical procedure. Another advantage is that a context-aware system for the surgical hub can improve surgical outcomes by adjusting surgical instruments (and other modular devices) for the specific context of each surgical procedure (e.g., for different tissue types) and verifying actions during the procedure. Yet another advantage is that the context-aware system can improve physician efficiency when performing surgical procedures by automatically suggesting the next steps, providing data, and adjusting displays and other modular devices in the operating room according to the specific context of the procedure. Another advantage is that the context-aware surgical hub can proactively and automatically control modular devices according to the specific step of the surgical procedure being performed, such as proactively activating a generator to which an RF electrosurgical instrument is connected if it determines that a subsequent step of the procedure requires the use of an instrument, thereby reducing the number of times the physician needs to interact with or control the surgical system during the surgical procedure. By actively activating the energy source, the device can be ready for use as soon as the preceding steps of the procedure are completed.

[0121] For example, a situation-aware surgical hub can be configured to determine what type of tissue is being operated on. Therefore, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the situation-aware surgical hub can be configured to precisely accelerate or decelerate the motor of the surgical instrument for that type of tissue, for example, by changing or triggering a change in at least one variable parameter of an algorithm for the surgical instrument relating to motor speed or torque.

[0122] In another embodiment, the type of tissue being operated on may influence the adjustments made to the compression rate and load threshold of the surgical stapler for specific interstitial space measurements. A context-aware surgical hub can be configured to infer whether the surgical procedure being performed is a thoracic or abdominal procedure, and the surgical hub can determine whether the tissue clamped by the end effector of the surgical stapler is lung tissue (in a thoracic procedure) or gastric tissue (in an abdominal procedure). The surgical hub can then be configured to cause adjustments to the compression rate and load threshold of the surgical stapler to be appropriate for that type of tissue, for example, by changing or causing a change in at least one variable parameter of the algorithm for the surgical stapler relating to the compression rate and load threshold.

[0123] In yet another embodiment, the type of body cavity being operated on during an insufflation procedure may affect the function of the smoke exhauster. The situation-aware surgical hub can be configured to determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and to determine the type of procedure. Since certain types of procedures are generally performed in specific body cavities, the surgical hub can be configured to control the motor speed of the smoke exhauster appropriately for the body cavity being operated on, for example, by changing or causing a change in at least one variable parameter of the algorithm for the smoke exhauster relating to the motor speed. Thus, the situation-aware surgical hub can provide a consistent amount of smoke exhaust for both thoracic and abdominal procedures.

[0124] In yet another embodiment, the type of procedure being performed may affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. A context-aware surgical hub can be configured to determine whether the surgical procedure is an arthroscopy. The surgical hub can be configured to compensate for the fluid-filled environment by adjusting the RF power level or ultrasonic amplitude of the generator (e.g., adjusting the energy level) by changing or triggering a change in at least one variable parameter of the algorithm for the instrument and / or generator relating to the energy level. Relatedly, the type of tissue being operated on may affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The context-aware surgical hub can be configured to determine what type of surgical procedure is being performed and then customize the energy levels for the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure by changing or triggering a change in at least one variable parameter of the algorithm for the instrument and / or generator relating to the energy level. Furthermore, the situation-aware surgical hub can be configured to adjust the energy levels of ultrasonic surgical instruments or RF electrosurgical instruments not simply for each surgery, but throughout the course of the surgical procedure. The situation-aware surgical hub can be configured to determine which stage of the surgical procedure is being performed or is continuing, and then update the control algorithms of the generator and / or ultrasonic surgical instruments or RF electrosurgical instruments to set the energy levels to values ​​appropriate for the expected tissue type according to the stage of the surgical procedure.

[0125] In another embodiment, the situational awareness surgical hub may be configured to determine whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display, according to the characteristics of the surgical site that the surgeon and / or physician are expected to need to see. The surgical hub may be configured to proactively change the displayed view (supplied, for example, by an imaging device for a visualization system) as appropriate, thereby automatically adjusting the display throughout the surgical procedure.

[0126] In yet another embodiment, the situation-aware surgical hub can be configured to determine which steps of a surgical procedure are being performed or will be performed, and whether specific data or comparisons of data are necessary for that step of the surgical procedure. The surgical hub can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon or physician to ask for specific information.

[0127] In another embodiment, a situation-aware surgical hub can be configured to determine whether a surgeon and / or other physician is making an error or deviating from an expected sequence of actions during the course of a surgical procedure, for example, as provided in a preoperative surgical plan. For example, the surgical hub can be configured to determine the type of surgical procedure being performed, read a corresponding list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the course of the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub has determined is being performed. The surgical hub can be configured to provide (visual, auditory, and / or tactile) alerts indicating that an unexpected action is being taken or an unexpected device is being used at a particular step in the surgical procedure.

[0128] In certain examples, the operation of a robotic surgical system, such as one of the various robotic surgical systems described herein, can be controlled by a surgical hub based on its situational awareness and / or feedback from its components, and / or information from the cloud (e.g., cloud 713 in Figure 18).

[0129] Embodiments of the situation recognition system and embodiments of using the situation recognition system during surgical procedures are described in the aforementioned U.S. Patent Application No. 16 / 729,772, titled "Analyzing Surgical Trends By A Surgical System," filed on December 30, 2019; U.S. Patent Application No. 16 / 729,747, titled "Dynamic Surgical Visualization Systems," filed on December 30, 2019; U.S. Patent Application No. 16 / 729,744, titled "Visualization Systems Using Structured Light," filed on December 30, 2019; U.S. Patent Application No. 16 / 729,778, titled "System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue," filed on December 30, 2019; and "Surgical Systems For Proposing And Corroborating Organ Portion." U.S. Patent Application No. 16 / 729,729, entitled "Removals", U.S. Patent Application No. 16 / 729,778, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ", filed on December 30, 2019, entitled "Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto", U.S. Patent Application No. 16 / 729,729, entitled "Removals", filed on December 30, 2019U.S. Patent Application No. 751, entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data" (16 / 729,740), filed December 30, 2019; U.S. Patent Application No. 16 / 729,737, entitled "Adaptive Surgical System Control According To Surgical Smoke Cloud Characteristics", filed December 30, 2019; U.S. Patent Application No. 16 / 729,796, entitled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics", filed December 30, 2019; U.S. Patent Application No. 16 / 729,803, entitled "Adaptive Visualization By A Surgical System", filed December 30, 2019; and "Method Of Using Imaging Devices In Further details are provided in U.S. Patent Application No. 16 / 729,807, entitled "Surgery."

[0130] Surgical system with hybrid intraluminal and extraluminal devices In certain embodiments, the surgical system is configured to allow one or more intraluminal instruments to be introduced into an organ using a laparoscopic approach. That is, unlike conventional systems (e.g., systems with intraluminal instruments introduced through natural orifices), this surgical system includes intraluminal instruments that, when used, approach and enter the organ (e.g., colon, bladder, stomach, etc.) through the laparoscopic side of the organ. This can provide two-handed operation capability with reduced technical complexity.

[0131] Furthermore, in some embodiments, laparoscopic instruments (e.g., grippers) and / or features (e.g., seals or stent-like structures) can be introduced into the extraluminal anatomical space and configured to provide local support for a portion (e.g., distal portion) of an intraluminal instrument. This local support can improve the intraluminal reactive loading capacity of the intraluminal instrument. Specifically, the local support can allow movement of the intraluminal instrument under load, enabling rotation, longitudinal advancement, and contact between the end effector (e.g., ablation element or jaw) of the intraluminal instrument and different intraluminal walls of the surgical site.

[0132] In one exemplary embodiment, the surgical system may generally include a first scope device having a first portion configured to be inserted and positioned in the extraluminal anatomical space, and a second portion located distal to the first portion and configured to be positioned in the intraluminal anatomical space, and a second instrument configured to be inserted in the extraluminal anatomical space and coupled to the first portion of the first scope device in the extraluminal anatomical space to facilitate the movement of the second portion of the first scope device while the second portion is positioned in the intraluminal anatomical space, and configured to move the first portion. In some embodiments, the first scope device may be a flexible body having a working channel extending through its interior, and a first imaging system at the distal end of the first scope device. The working channel is configured to allow the distal end of the first instrument to be inserted into the extraluminal anatomical space and then inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal and intraluminal spaces.

[0133] During use, generally, the first part of the first device scope is inserted into the extraluminal anatomical space, and the second part of the first scope device is further inserted into the intraluminal anatomical space. Then, the first instrument is inserted through the working channel, positioning the first instrument in both the extraluminal and intraluminal spaces. Furthermore, the second instrument is inserted into the extraluminal anatomical space. The second instrument can be inserted into the extraluminal anatomical space before, simultaneously with, or after the insertion of the first device scope or the first instrument. After insertion, the second instrument is moved to move the inserted second part of the first scope device within the intraluminal anatomical space. Before the insertion of any one of the first scope device, the first instrument, or the second instrument, the extraluminal space, the intraluminal space, or both can be ventilated, for example, through a fluid port operably coupled to the first part of the first scope device.

[0134] In another exemplary embodiment, the surgical system may generally include: an anchor member positioned in an extraluminal anatomical space and configured to contact a tissue wall that at least partially defines an intraluminal anatomical space; a cannula having a first portion configured to be inserted and positioned in the extraluminal anatomical space and a second portion located distal to the first portion and configured to be positioned in the intraluminal anatomical space; and a selectively deployable stabilizing member positioned on the first portion of the cannula in the extraluminal anatomical space and configured to connect to the anchor member. In some embodiments, the cannula may be configured such that the distal end of the first instrument is inserted into the extraluminal anatomical space and then inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal and intraluminal anatomical spaces. Furthermore, the selectively deployable stabilizing member may be configured to facilitate the rotational movement of the first instrument in the intraluminal anatomical space by providing an anchor point for the first instrument when deployed.

[0135] Exemplary surgical systems, as described herein and shown in the drawings, may have a variety of features. However, it will be recognized by those skilled in the art that surgical systems may have only some of these features and / or various other features well known in the art. The surgical systems described herein are intended to represent specific exemplary embodiments only. Furthermore, although surgical systems are shown and described in relation to the colon, those skilled in the art will understand that these surgical anchoring systems may be used in relation to any other suitable body cavities or organs.

[0136] Surgical resection of partial tissue wall tumors is conventionally performed through a natural opening. For example, as shown in Figure 22, the colon 10000 contains a partial tissue wall tumor 10001. As shown, a conventional surgical system includes an endoscope 10004 inserted into the colon 10000 through the rectum 10002, and a first instrument 10008 inserted through the working channel 10006 of the endoscope 10004. The first instrument 10008 engages with the tumor 10001 for subsequent removal. Laparoscopic instruments 10010 (e.g., grippers) are inserted through the abdominal cavity 10012 and interact with the colon 10000 to assist in stabilizing the tumor 10001 or positioning the colon 10000 for tumor removal. Unlike these conventional surgical systems and procedures, as will be discussed in more detail below, the surgical system disclosed herein is designed to remove affected tissue (e.g., lesion or tumor) using intraluminal instruments that access the natural lumen or organ from the laparoscopic side, rather than through the natural opening.

[0137] Figure 23 illustrates an exemplary embodiment of surgical system 10100, which includes a first scope device 10102 and first and second laparoscopic instruments 10104 and 10106, used in the surgical resection of a partial tissue wall thickening tumor 10101 located within the colon 10103. For simplicity, specific components of surgical system 10010 are not shown.

[0138] Each of the first laparoscopic instrument 10104 and the second laparoscopic instrument 10106 is inserted into the abdominal cavity 10105 (e.g., extraluminal anatomical space) through the respective first trocar 10108 and second trocar 10110. Each of the first trocar 10108 and the second trocar 10110 is coupled to the respective robotic arms 10112 and 10114. The first laparoscopic instrument 10104 and the second laparoscopic instrument 10106 can have various configurations, but in this embodiment, each laparoscopic instrument 10104, 10106 has an elongated shaft 10104a, 10106a with end effectors 10104b, 10106b at its distal end. Each end effector 10104b, 10106b can have various configurations, but in this illustrated embodiment, each end effector 10104b, 10106b is in the form of a set of movable jaws. Furthermore, although two laparoscopic instruments are shown, in other embodiments any number of laparoscopic instruments (e.g., one, three, four, etc.) can be used.

[0139] The first scope device 10102 includes a flexible body 10116 having a working channel 10118 extending through its interior, and a first imaging system 10120 (e.g., a camera) at its distal end. The flexible body can be formed from any suitable flexible material(s). As shown, in use, the proximal end of the first scope device 10102 is connected to the first robotic arm 10120, and the first scope device 10102 extends into and through a trocar 10122 connected to a second robotic arm 10124 into the abdominal cavity 10105 (e.g., extraluminal anatomical space). The trocar 10122 includes a fluid port 10123 configured to ventilate the abdominal cavity 10105 prior to or simultaneously with the insertion of any device or instrument into the abdominal cavity 10105. In other embodiments, the abdominal cavity 10105 can be ventilated using trocars 10112, 10114, or any other suitable ventilation mechanisms and devices.

[0140] The first scope device 10102 is further inserted into the colonic lumen 10107 (e.g., intraluminal anatomical space) through the wall 10103a of the colon 10103. The first scope device 10102 can be inserted directly through an incision 10115 made in the colonic wall 10103a, but in this illustrated embodiment, the lumen of the cannula 10117 is partially inserted into the colonic lumen 10107 through the incision 10115. Thus, the first scope device 10102 is inserted into the colonic lumen 10107 through the lumen of the cannula 10117.

[0141] As shown, the first scope device 10102 has a first portion 10102a located in the abdominal cavity 10105 and a second portion 10102b located distal to the first portion 10102a and located in the colonic lumen 10107. That is, the first scope device 10102 is designed to be introduced into the colon 10103 by a laparoscopic approach. Before inserting the second portion of the first scope device, the colon can be ventilated, for example, by introducing fluid through a fluid port (not shown) or lumen (not shown) that has been previously inserted into the colon. After ventilation, sealing clips 10109a and 10109b can be positioned on the opposing ends of the ventilated area of ​​the colon 10103.

[0142] In some embodiments, the trocar 10122 can provide structural support for the first portion 10102a of the first scope device 10102. Furthermore, the first portion 10102a of the first scope device 10102 may be driven from one or more tool drivers (not shown) located within a motor housing 10121 positioned between the robot arm 10124 and the trocar 10122.

[0143] Since the first scope device 10102 has a flexible body 10116 that is present in both the abdominal cavity 10105 and the colonic cavity 10107, a cooperative support element is required so that the second portion 10102b of the first scope device 10102 can move within the colonic cavity 10107. In this illustrated embodiment, the cooperative support element is the first laparoscopic instrument 10104. That is, as shown, the jaws of the end effector 10104b grasp the first portion 10102a of the first scope device 10102, thus connecting the first laparoscopic instrument 10104 to the first portion 10102a of the first scope device 10102.

[0144] The jaws of the end effector can grasp the first portion 10102a of the first scope device 10104 in various positions, but in this illustrated embodiment, the first laparoscopic instrument 10104 is located in the abdominal cavity 10105 (e.g., extraluminal anatomical space) and is coupled to the first portion 10102 at a predetermined position directly adjacent to the colon wall 10103a. More specifically, the predetermined position is close to an incision 10115 made in the colon wall 10103a. In this embodiment, the elongated shaft 10104a of the first laparoscopic instrument 10104 is rigid and therefore provides support to the first scope device, allowing the first portion of the first scope device to move within the abdominal cavity (e.g., extraluminal space) and facilitating the movement of the second portion of the first scope device 10102 within the colon cavity 10107 (e.g., intraluminal anatomical space).

[0145] In some embodiments, the fixation provided by the first laparoscopic instrument 10104 can keep the incision 10115 upright to prevent leakage of colonic contents into the abdominal cavity 10105. Alternatively, or in addition, the jaws of the end effector 10104b can be configured to act as a wound protector, which can prevent the first scope device 10102 from applying an inappropriate load to the incision margin.

[0146] As further shown in Figure 23, once the second portion 10102b of the first scope device 10102 is positioned within the colonic lumen 10107, the instrument 10126 can be inserted into the colonic lumen 10107 through the working channel 10118 of the first scope device 10102. Once inserted, the instrument 10126 can interact with the tumor 10102 for subsequent removal. Furthermore, the jaws of the end effector 10106b of the second laparoscope device 10106 can interact with the colon 10103 to help stabilize the colon 10103 for removal of the tumor 10102.

[0147] In some embodiments, localized mechanical docking can be used as a mechanism for stabilizing a flexible intraluminal device or instrument, or device, as illustrated in Figure 24, for example.

[0148] Figure 24 illustrates an exemplary embodiment of surgical system 10200 configured to allow laparoscopic access to an intraluminal surgical site. The surgical system 10200 includes a cannula 10202, an anchor member 10204, and a selectively deployable stabilizing member 10206, which are used in the surgical resection of a partial tissue wall tumor 10201 located within the colon 10203.

[0149] The cannula 10202 can have a variety of different configurations. In this illustrated embodiment, the cannula 10202 has a first portion 10202a configured to be inserted and positioned within the abdominal cavity 10208 (e.g., extraluminal anatomical space), and a second portion 10202b distal to the first portion 10202a configured to be positioned within the lumen 10210 of the colon 10203 (e.g., intraluminal anatomical space). The cannula 10202 can be formed from any suitable material. As shown in the illustration, during use, the cannula 10202 is inserted into the abdominal cavity 10208 through a trocar 10212 coupled to a robotic arm 10214. The trocar 10212 includes a fluid port 10216 configured to ventilate the abdominal cavity 10208 prior to or simultaneously with the insertion of any device or instrument into the abdominal cavity 10208. In other embodiments, the abdominal cavity 10208 can be ventilated using another trocar or any other suitable ventilation mechanism and device.

[0150] The cannula 10202 is further inserted into the colonic lumen 10210 (e.g., intraluminal anatomical space) through the wall 10205 of the colon 10203. Thus, the cannula 10202 is designed to be introduced into the colon 10203 by a laparoscopic approach. Furthermore, once the first portion 10202a and the second portion 10202b are positioned in the abdominal cavity 10208 and the colonic lumen 10201, respectively, the first instrument 10220 may be inserted through them so that the distal end of the first instrument 10220 is positioned in the colonic lumen 10210 and can be used to remove the tumor 10201.

[0151] As shown, the cannula 10202 allows the distal end of the first instrument 10220 to be introduced into the colon 10203 through the abdominal cavity 10208, so that the first instrument 10220 is present in both the abdominal cavity 10208 and the colonic cavity 10210. The first instrument 10220 can have various configurations, but in this illustrated embodiment, the first instrument 10220 includes a flexible shaft 10222 having a pair of jaws 10224 at its distal end. The pair of jaws 10224 are configured to interact with the tumor 10201.

[0152] Before inserting the second portion 10202b of the cannula 10202, the colon can be ventilated by introducing fluid, for example, through a fluid port (not shown) or lumen (not shown) that has been previously inserted into the colon 10203. After ventilation, the ventilated area 10203a can be sealed. For example, in this illustrated embodiment, the ventilated area 10203a is sealed by the jaws 10218a and 10218b of a laparoscopic device 10218 inserted into the abdominal cavity 10208, with the jaws 10218a and 10218b gripping one end of the area, and by an anchor member 10204 clipped around the opposite end of the area. Thus, in this illustrated embodiment, the anchor member 10204 can function as both an anchor and a seal. In other embodiments, separate sealing elements may be used.

[0153] The connecting member 10204 can have various configurations. In this illustrated embodiment, the anchor member 10202 is in the form of a clip positioned within the abdominal cavity (e.g., extraluminal anatomical space) and in contact with the outer surface of the tissue wall 10203a of the colon 10203. Prior to, simultaneously with, or subsequently to the insertion of the cannula 10202, the anchor member 10204 can be inserted into the abdominal cavity 10208 and positioned in contact with the colon wall 10205 (e.g., arranged around a portion of the colon 10203).

[0154] As further shown in Figure 24, the selectively deployable stabilizing member 10206 is positioned on the first portion 10202a of the cannula 10202 and therefore within the abdominal cavity 10208 (e.g., extraluminal anatomical space). The selectively deployable stabilizing member 10206 can have various configurations. In this illustrated embodiment, the selectively deployable stabilizing member 10206 includes a first link 10223a and a second link 10223b pivotably connected to each other, the first link 10223a being directly coupled to the cannula 10202. Thus, the selectively deployable stabilizing member 10206 can move from a non-deployed state to a deployed state (Figure 24).

[0155] When in use, the selectively deployable stabilizing member 10206, when in the deployed state (Figure 24), is configured to connect to the anchor member 10204. This connection provides an anchor point for the first instrument 10220, which is inserted through the cannula 10202. Since the first instrument 10220 includes a flexible shaft 10222, the anchor point allows the first instrument 10220 to pivot within the colonic lumen 10210 relative to the cannula 10202.

[0156] The selectively deployable stabilizing member 10206 can be coupled to the anchor member 10204 in a variety of ways. For example, in a particular embodiment, the anchor member 10204 may include a magnet 10226 configured to couple the selectively deployable stabilizing member 10206 to the anchor member 10204 when the selectively deployable stabilizing member 10206 is in the deployed state. In other embodiments, any other suitable coupling mechanism may be used.

[0157] Furthermore, additional instruments or devices can be inserted through the cannula 10202 (for example, through one or more luminal parts of the cannula). For example, as shown in Figure 24, the first scope device 10228 can be inserted into and through the cannula 10202 such that the first part of the first scope device 10228 is located in the abdominal cavity (e.g., extraluminal anatomical space) and the second part of the first scope device 10228, distal to the first part, is located in the colonic lumen (e.g., intraluminal anatomical space).

[0158] In other embodiments, a robot-operable and lockable cannula can be used to introduce an intraluminal instrument into an intraluminal anatomical space using a laparoscopic approach. For example, as shown in Figure 25, a robot-operable and lockable cannula 10300 can be inserted into an extraluminal anatomical space 10304 (e.g., the abdominal cavity) through a first trocar 10302. The first trocar 10302 is coupled to a first robotic arm 10306. As further shown, a first instrument 10308 may be coupled to a second robotic arm 10310 and inserted through the first trocar 10302. The first instrument 10308 may be further inserted through the robot-operable and lockable cannula 10300 such that the distal end 10310 of the first instrument 10308 extends through the distal end 10312 of the cannula 10300. As a result, the first instrument 10308 is structurally guided and supported by the maneuverable and lockable distal end 10302 of the cannula 10300. Movement of the cannula 10300 (e.g. by the first robotic arm 10306) can therefore guide the distal end 10312 of the first instrument 10308 into the organ cavity 10318 through the incision 10314 made in the organ 10316, and movement of the second robotic arm 10310 can move the distal end 10312 of the first instrument 10308 within the organ cavity 10318 relative to the distal end 10312 of the cannula 10300.

[0159] Surgical system with collaborative instruments inside and outside the lumen The devices, systems, and methods for multi-source imaging provided herein enable collaborative surgical visualization. Generally, in collaborative surgical visualization, a first imaging system and a second imaging system (e.g., a first scope device and a second scope device), each acquiring images of the surgical site, are configured to cooperate to provide improved imaging of the surgical site. Collaborative surgical visualization can improve the visualization of the patient's anatomical structures at the surgical site and / or improve the control of the surgical instrument(s) at the surgical site.

[0160] In certain embodiments, the surgical system is configured to be positioned within two separate anatomical regions to perform one or more surgical tasks. The surgical visualization system can enable intraoperative identification of critical structures (e.g., affected tissue, anatomical structures, surgical instruments, etc.). Thus, the surgical visualization system can enable improved surgical decision-making and improved surgical outcomes. The surgical visualization system can provide advanced visualization capabilities beyond what the physician sees with the "naked eye" and / or what the imaging system can perceive and / or communicate to the physician. The surgical visualization system can reinforce and enhance what the physician can know before tissue treatment (e.g., incision), thereby improving outcomes in various cases. As a result, the physician can ensure that they maintain momentum throughout the surgical procedure, knowing, for example, that the surgical visualization system is tracking critical structures that may be approached during incision. The surgical visualization system can provide instructions to the physician with sufficient time to pause and / or slow down the surgical procedure and assess the proximity to critical structures to prevent inadvertent damage to those structures. A surgical visualization system provides physicians with an ideal, optimized, and / or customizable amount of information, enabling them to move confidently and / or quickly throughout the tissue while avoiding accidental damage to healthy tissue and / or critical structures, thereby minimizing the risk of injury resulting from surgical procedures.

[0161] Generally, the surgical systems provided herein generally include: a first scope device positioned in both intraluminal and extraluminal anatomical spaces and configured to transmit image data of a first scene within its field of view; a second scope device inserted into the extraluminal anatomical space and configured to transmit image data of a second, different scene within its field of view; and a controller configured to receive the transmitted data, determine the relative distance between the first and second scope devices, and provide a merged image. The merged image may be at least a portion of at least the first and second scope devices within a single scene, and at least one of the first and second scope devices within the merged image is a representative depiction thereof. Thus, the merged image can therefore provide two distinct viewpoints of the surgical site, which can conveniently allow the physician to view only one display instead of multiple displays. Furthermore, within that single display, the merged images allow the physician to adjust the relative position and / or orientation of at least the first device and scope device positioned at or near the surgical site. In one embodiment, the surgical system may include a tracking device associated with one of the first or second scope device and configured to transmit a signal indicating the location of one of the first or second scope device relative to the other of the first or second scope device.

[0162] The surgical systems provided herein can also be used in various robotic surgical systems as described above, and can incorporate various tracking and / or imaging mechanisms, such as electromagnetic (EM) tracking chips, fiber Bragg gratings, virtual tags, reference markers, probes, identification of known anatomical structures, various 3D scanning techniques such as using structured light, and various sensors and / or imaging systems described above, to assist in tracking the movement of instruments, endoscopes, and laparoscopes relative to each other and / or the system as a whole. The tracking mechanisms can be configured to transmit tracking data from both the laparoscope and the endoscope, thereby enabling the position of one scope relative to the other. Furthermore, important structures within the field of view of either scope (e.g., lesion tissue, surgical instruments, anatomical structures) can be tracked by the scope that has such important structures within its field of view. Overall, the surgical systems herein can track objects within the field of view of each scope and the relative positions of each scope. Thus, with the whole of the tracking data, the system can calculate the distance of important structures from scopes that do not have important structures within their field of view, based on the tracking data collected by the other scope.

[0163] In one exemplary embodiment, the surgical system also includes a first instrument and a second instrument. The first instrument is configured to be inserted into and through the extraluminal anatomical space into the intraluminal anatomical space, such that the first instrument is present in both the extraluminal and intraluminal anatomical spaces. The second instrument is configured to be inserted into the extraluminal anatomical space.

[0164] Furthermore, in some embodiments, an imaging system (e.g., a camera) can be positioned on a second portion of the first scope device and configured to transmit image data of the scene within the field of view of the first scope device. Alternatively or additionally, an imaging system (e.g., a camera) can be positioned on a second scope device and configured to transmit image data of the scene within the field of view of the second scope device. This enables coordinated visualization between instruments operating in extraluminal anatomical spaces and instruments operating in intraluminal anatomical spaces, and further enables instruments to operate coordinately together on a single surgical site.

[0165] In various embodiments, the surgical systems provided herein include a controller. The surgical systems, controllers, displays, and / or various instruments, endoscopes, and laparoscopes may also be incorporated into several different robotic surgical systems and / or be part of a surgical hub such as any of the systems and surgical hubs discussed above. The controller is generally configured to merge a first scene and a second scene from the endoscope and laparoscope, respectively, to visually form a merged image between the first and second scenes. The controller is configured to receive the tracking data detailed above and combine it with the first and second scenes to generate a merged image that includes at least a representative depiction of the endoscope or laparoscope and any structures within the scope's field of view that are visually obscured by tissue walls. For example, if the merged image is from the viewpoint of the endoscope, the merged image is a live image stream of what the endoscope is seeing, including an overlay of laparoscopically positioned surgical instruments and, if present, the orientation and position of the laparoscope.

[0166] During use, generally, the first part of the first scope device scope is inserted into the extraluminal anatomical space, and the second part of the first scope device (e.g., the distal part of the first part) is inserted into the intraluminal anatomical space. Furthermore, the second scope device is inserted into the extraluminal anatomical space. Furthermore, the first instrument is inserted into the extraluminal anatomical space and through the extraluminal anatomical space into the intraluminal anatomical space so that the first instrument is present in both the extraluminal and intraluminal anatomical spaces. For example, the first instrument can be inserted through the working channel of the first scope device, positioning the first instrument in both spaces. Furthermore, the second instrument is inserted into the extraluminal anatomical space. The second instrument can be inserted into the extraluminal anatomical space before, simultaneously with, or after the insertion of the first device scope or the first instrument.

[0167] Exemplary surgical systems, as described herein and shown in the drawings, may have a variety of features. However, it will be recognized by those skilled in the art that surgical systems may have only some of these features and / or various other features well known in the art. The surgical systems described herein are intended to represent specific exemplary embodiments only. Furthermore, although surgical systems are shown and described in relation to the colon, those skilled in the art will understand that these surgical systems may be used in relation to any other suitable body cavities or organs.

[0168] Figure 26 illustrates an exemplary embodiment of a surgical system 10400 having a first scope device 10412 and a second scope device 10414, used in the surgical resection of a partial tissue wall thickening tumor 10401 located within the colon 10402. For brevity, specific components of the surgical system 10400 are not shown.

[0169] The first scope device 10412 includes a flexible body 10422 through which a first working channel 10424 and a second working channel 10426 extend, and a first imaging system 10428 (e.g., a camera) at its distal end. The flexible body 10422 can be formed from any suitable flexible material(s).

[0170] During use, the proximal end of the first scope device 10412 is coupled to the first robotic arm 10430, and the first instrument 10432 is coupled to the second robotic arm 10434. The first scope device 10412 is inserted into the abdominal cavity 10405 (e.g., extraluminal anatomical space) through the first trocar 10436. The first trocar 10436 is coupled to the robotic arm 10438. The first scope device 10412 is further inserted into the colonic lumen 10407 (e.g., intraluminal anatomical space) through the lumen of the sealing port 10440, with the sealing port 10440 positioned within the wall 10406 ​​of the colon 10402. The first scope device 10412 can be inserted into and through the first trocar 10436 and sealing port 10440 such that the first portion 10412a of the first scope device 10412 is located in the abdominal cavity 10405 (e.g., extraluminal anatomical space) and the second portion 10412b of the first scope device 10412, distal to the first portion 10412a, is positioned in the colonic lumen 10407 (e.g., intraluminal anatomical space). In some embodiments, the sealing port 10440 can be omitted so that the first scope device 10412 is inserted directly through an incision made in the colonic wall 10402a.

[0171] As shown, the first scope device 10412 has a first portion 10412a located in the abdominal cavity 10405 and a second portion 10412b located distal to the first portion 10412a and located in the colonic cavity 10407. That is, the first scope device 10412 is designed to be introduced into the colon 10402 by a laparoscopic approach. Before or after insertion of the second portion 10412b of the first scope device 10412, sealing clips 10409a and 10409b can be positioned on the opposing ends of the ventilated region of the colon 10402.

[0172] In some embodiments, the first portion 10412a of the first scope device 10412 may be driven from one or more tool drivers (not shown) located in a motor housing 10421, positioned between the robot arm 10438 and the first trocar 10436.

[0173] As further shown in Figure 26, once the second portion 10412b of the first scope device 10412 is positioned within the colonic lumen 10407, the first instrument 10432 can be inserted through the first working channel 10424 of the first scope device 10412 so that the distal end of the first instrument is positioned within the colonic lumen 10407. As a result, the first instrument 10432 is present in both the abdominal cavity (e.g., extraluminal anatomical space) and the colonic lumen 10407 (e.g., intraluminal anatomical space). Once inserted, the end effector 10433 (of the first instrument 10432) can interact with the tumor 10401 for subsequent removal. The end effector 10433 can have various configurations, but in this illustrated embodiment, the end effector 10433 is in the form of a pair of movable jaws. In some embodiments, at least one of the first working channel 10424 and the second working channel 10426 is configured to allow instrument replacement without impairing the position of the first scope device 10412 in at least one of the abdominal cavity 10405 and the colon cavity 10407. This also allows for maintaining the field of view of the first imaging system 10428.

[0174] The surgical system 10400 also includes a controller 10470 communicatively coupled to the endoscope 10412 and the laparoscope 10414, and configured to receive transmitted image data of a first scene and a second scene from a first optical sensor 10428 and a second optical sensor 10458, respectively. In some embodiments, the controller 10470 is also communicatively coupled to a first tracking device 10482 and a second tracking device 10484 located in the endoscope 10412 and the laparoscope 10414, respectively, and configured to receive signals transmitted from the first and second tracking devices. Upon reception, the controller 10470 is configured to determine at least the relative distance between the endoscope 10412 and the laparoscope 10414. In some embodiments, the controller 10470 may also be configured to determine the relative orientation between the endoscope 10412 and the laparoscope 10414.

[0175] As further shown in Figure 26, the second scope device 10414 is positioned laparoscopically within the abdominal cavity 10405. The second scope device includes a flexible body 10452 through which third and fourth working channels 10454, 10456 extend, and a second imaging system 10458 (e.g., a camera) at its distal end. The flexible body 10452 can be formed from any preferred flexible material(s).

[0176] The second scope device 10414 is inserted into the abdominal cavity 10405 (e.g., extraluminal anatomical space) through a second trocar 10466 positioned within the abdominal wall 10403. The second trocar 10466 is coupled to the second robotic arm 10468. The second scope device 10414 is inserted and positioned within the abdominal cavity 10405 (e.g., extraluminal anatomical space). In some embodiments, the flexible body 10452 of the second scope device 10414 can be driven from one or more tool drivers (not shown) in a motor housing 10451 positioned between the second robotic arm 10468 and the second trocar 10466.

[0177] As shown, during use, the proximal end of the second scope device 10414 is coupled to the first robotic arm 10460, and the second instrument 10462 is coupled to the second robotic arm 10464. The second instrument 10462 extends into and through the third working channel 10454 into the abdominal cavity 10405 (e.g., extraluminal anatomical space). The second instrument 10462 can have various configurations, but in this illustrated embodiment, the second instrument 10462 has an elongated shaft 10462a with an end effector 10463 at its distal end. In some embodiments, the second instrument 10462 is configured to assist in manipulating the colon 10402 from the abdominal cavity 10405 (e.g., extraluminal anatomical space) in order to position the first instrument 10432 within the colonic lumen 10407 (e.g., intraluminal anatomical space). Furthermore, the end effector 10463 of the second instrument 10462 can interact with the colon 10402 to help stabilize the colon 10402 for the removal of the tumor 10401. The end effector 10463 can have various configurations, but in this illustrated embodiment, the end effector 10463 is in the form of a pair of movable jaws. In some embodiments, the end effector 10463 can be used to form a seal within the colonic lumen 10407 (for example, by clamping the colon 10402).

[0178] As shown in Figure 26, the reference marker 10480 can be positioned on the first portion 10412a of the endoscope 10412. The reference marker 10480 is within the field of view of the optical sensor 10458 of the laparoscope 10414. The reference marker 10480 is fixed on the outer surface of the first portion of the endoscope 10412 so that the position of the second portion 10412b of the endoscope 10412 can be determined through visualization of the reference marker 10480 by the optical sensor 10458. Based on both the image data transmitted from the optical sensor 10458 that identifies the reference marker 10480, the controller 10470 is configured to provide a merged image on, for example, the first display 10471, the second display 10472, or both displays of the surgical system 10400. In the merged image, at least one of the endoscope 10412 and the laparoscope 10414 is a representative depiction. Various embodiments of magnetic reference markers, and the use of magnetic reference markers in detecting position, are discussed further, for example, in U.S. Provisional Patent Application No. 63 / 249,658, filed September 29, 2021, entitled "Surgical Devices, Systems, and Methods for Control of One Visualization with Another".

[0179] In some embodiments, the reference marker is a visually identifiable physical symbol. In other embodiments, the reference marker may be a light-emitting device or an electromagnetic radiation device that can be identified by a laparoscope to track the endoscope. In addition, there may be multiple reference markers positioned on the outer surface of a first portion of the endoscope, and the laparoscope's optical sensor can identify which reference marker is in the extraluminal space.

[0180] The first display 10471 and the second display 10472 can have various configurations. For example, in some embodiments, the first display may be configured to display a first scene, the second display may be configured to display a second scene, and the first display, the second display, or both may be further configured to display a merged image. In another embodiment, the surgical system 10400 may include a third display 10473 that can be used to display a merged image, and the first display 10471 and the second display 10472 are used solely to display image data transmitted from the optical sensors 10428 and 10458 without modification, respectively. In this embodiment, the surgeon can access real-time scenes from both the endoscope 10412 and the laparoscope 10414 on the first display 10471 and the second display 10472, while also having access to a merged image on the third display 10473.

[0181] As described above, the endoscope 10412 includes a first optical sensor 10428. The first optical sensor 10428 is configured to transmit image data of a first scene within the field of view of the endoscope 10412 to the controller 10470. In this illustrated embodiment, the tumor 10401 and surgical instruments 10432 are positioned within the field of view of the endoscope 10412. In some embodiments, the relative distance between the endoscope 10412 and the laparoscope 10414 can be determined by using structured light projected onto a first portion 10412a and a reference marker 10480 (e.g., via an illumination element) and tracked by a second optical sensor 10458. Furthermore, in some embodiments, based on the determined relative distance between the endoscope 10412 and the laparoscope 10414, and the determined relative distance between the endoscope 10412 and the tumor 10401, the controller can calculate the relative distance between the laparoscope 10414 and the tumor 10401.

[0182] In addition, the laparoscope 10414 includes a second optical sensor 10458. The second optical sensor 10458 is configured to transmit image data of a second scene within the field of view of the laparoscope 10414 to the controller 10470. The surgical instrument 10462 is positioned within the field of view of the laparoscope 10414. As a result, the controller 10470 can determine the relative distance between the surgical instrument 10462 and the surgical instrument 10432 based on the transmitted image data.

[0183] Figure 26a shows an exemplary embodiment of a merged image. The merged image illustrates a real-time second scene within the field of view of the laparoscope 10414 and an overlaid representative depiction of a portion of the colon 10402 on the endoscopic side (e.g., tumor 10401 and / or endoscope 10412). Based on the transmitted image data of the second scene combined with a reference marker 10480, the controller 10470 can provide a merged image from the viewpoint of the laparoscope 10414, where endoscope 10412 and tumor 10401 are shown as representative depictions corresponding to their positions in the intraluminal space in real time. In the illustrated embodiment, the representative depictions are indicated by dashed outlines of the corresponding tumor 10401 and endoscope 10412. However, other forms of representative depictions can be used, such as simple geometric shapes to represent non-visual instruments and anatomical structures in the intraluminal space.

[0184] Alternatively, or in addition, the controller 10470 can generate merged images from the viewpoint of the endoscope 10412. For example, in Figure 26b, the merged image shows a real-time first scene in the field of view of the endoscope 10412 with an overlaid representative depiction of a portion of the laparoscopic side of the colon 10402 (e.g., the laparoscope 10414 and / or surgical instrument 10462). Those skilled in the art will understand that the term “representative depiction” as used herein refers to a virtual overlay on the actual depiction from the camera, where the virtual overlay corresponds to the position and orientation of an object that is located within the camera’s field of view but is not visible to the camera due to an obstruction located between the camera and the object, and that the term “actual depiction” as used herein refers to an unmodified real-time image or video stream from the camera. Based on the transmitted image data from the optical sensor 10428 combined with the reference marker 10480, the controller 10470 can provide a merged image from the viewpoint of the endoscope 10412, where the laparoscope 10414 and surgical instruments 10462 are shown in real time as representative depictions corresponding to their locations in the extraluminal space. In the illustrated embodiment, the representative depictions are indicated by dashed outlines of the laparoscope 10414 and surgical instruments 10462. However, other forms of representative depictions can be used, such as simple geometric shapes to represent non-visual instruments and anatomical structures in the intraluminal space.

[0185] In one embodiment, the movement of instruments in both the intraluminal and extraluminal spaces can be coordinated, as instruments in both sets can be visualized by the other. For example, cooperative defect repair (e.g., suturing an incision) can be achieved by inserting a needle hook from the laparoscopic side using an instrument, and then passing the needle hook into the intraluminal space, where an endoscopically arranged instrument can grasp the hook needle. The hook needle can then be returned to the extraluminal space through the colon, and this process is repeated until the incision is sutured closed.

[0186] In other embodiments, the positions of the endoscope and laparoscope can be tracked relative to each other through a time-based approach. The point in time when the scope devices can no longer visually distinguish each other can be used as a reference point. The movement of each scope device by a robotic arm can be recorded, and the position of each scope device can be determined over time as the scope devices are moved within anatomical space.

[0187] A surgical system for independent ventilation of two separate anatomical spaces. In certain embodiments, the surgical system is configured to ventilate two separate anatomical regions for performing one or more surgical tasks. Generally, the surgical system includes a first access port(s) configured to provide access to a first cavity (e.g., an extraluminal anatomical space) and allow ventilation thereof, and a second access port(s) configured to provide access to a separate cavity (e.g., an intraluminal anatomical space) through the first cavity and allow ventilation thereof. This can provide separate anatomical working volumes for different instruments and further allow these different instruments to work together on a single surgical site.

[0188] In one exemplary embodiment, the surgical system generally includes a first scope device configured to be positioned in both the intraluminal and extraluminal anatomical spaces, and a second scope device configured to be inserted into the extraluminal anatomical space. The first scope device has a first ventilation port (e.g., a fluid port) operably coupled to the first scope device and configured to vent the intraluminal anatomical space into the first ventilation space, and the second scope device has a second ventilation port (e.g., a fluid port) operably coupled to the second scope device and configured to vent the extraluminal anatomical space into the second ventilation space. Thus, both the first and second ventilation spaces are independently pressurized and thus generated to provide separate working volumes for different instruments.

[0189] The surgical system also includes a first instrument and a second instrument. The first instrument is configured to be inserted into the extraluminal anatomical space and through the extraluminal anatomical space into the intraluminal anatomical space, such that the first instrument is present in both the extraluminal and intraluminal anatomical spaces. The second instrument is configured to be inserted into the extraluminal anatomical space.

[0190] In some embodiments, the surgical system may include a sealing port located in the tissue wall that separates the extraluminal anatomical space from the intraluminal anatomical space. In certain embodiments, the sealing port is configured to allow a second portion of a first scope device to enter the intraluminal anatomical space.

[0191] Furthermore, in some embodiments, an imaging system (e.g., a camera) can be positioned on a second portion of a first scope device and configured to transmit image data of the scene within the field of view of the first scope device. Alternatively or additionally, an imaging system (e.g., a camera) can be positioned on a second scope device and configured to transmit image data of the scene within the field of view of the second scope device. This enables coordinated visualization between instruments operating in extraluminal anatomical spaces and instruments operating in intraluminal anatomical spaces, and further enables instruments to work together coordinately on a single surgical site. Furthermore, coordinated visualization can be used when adjustments may need to be made to the first ventilation area, the second ventilation space, or both during a particular surgical task or step or the entire surgical procedure. The imaging system may include multiple cameras, which the surgeon can use to achieve a better view of the surgical site within the patient's body.

[0192] During use, generally, the first part of the first scope device scope is inserted into the extraluminal anatomical space, and the second part of the first scope device (e.g., the distal part of the first part) is inserted into the intraluminal anatomical space. Furthermore, the second scope device is inserted into the extraluminal anatomical space. Before, simultaneously with, or after the insertion of the first scope device, the intraluminal anatomical space can be vented to a first pressure using the first ventilation port, thereby forming the first ventilation space. Furthermore, before, simultaneously with, or after the insertion of the first device scope, the intraluminal anatomical space, and / or the insertion of the second device scope, the extraluminal anatomical space can be vented to a second pressure via the second ventilation port, thereby forming the second ventilation space.

[0193] Furthermore, the first instrument is inserted into the extraluminal anatomical space and through the extraluminal anatomical space into the intraluminal anatomical space, such that the first instrument is present in both the extraluminal and intraluminal anatomical spaces. For example, the first instrument can be inserted through the working channel of the first scope device, positioning the first instrument in both spaces. The first instrument can be inserted before, simultaneously with, or after ventilation of the intraluminal anatomical space, the extraluminal space, or both. Furthermore, a second instrument is inserted into the extraluminal anatomical space. The second instrument can be inserted into the extraluminal anatomical space before, simultaneously with, or after, the insertion of the first device scope, the insertion of the first instrument, ventilation of the intraluminal anatomical space, or ventilation of the extraluminal space.

[0194] Exemplary surgical systems, as described herein and shown in the drawings, may have a variety of features. However, it will be recognized by those skilled in the art that surgical systems may have only some of these features and / or various other features well known in the art. The surgical systems described herein are intended to represent specific exemplary embodiments only. Furthermore, although surgical systems are shown and described in relation to the colon, those skilled in the art will understand that these surgical systems may be used in relation to any other suitable body cavities or organs.

[0195] Figure 27 illustrates an exemplary embodiment of a surgical system 10500 having a first scope device 10512 and a second scope device 10514, used in the surgical resection of a partial tissue wall thickening tumor 10501 located within the colon 10502. For brevity, specific components of the surgical system 10500 are not shown.

[0196] The first scope device 10512 includes a flexible body 10522 through which a first working channel 10524 and a second working channel 10526 extend, and a first imaging system 10528 (e.g., a camera) at its distal end. The flexible body 10522 can be formed from any suitable flexible material(s).

[0197] During use, the proximal end of the first scope device 10512 is coupled to the first robotic arm 10530, and the first instrument 10532 is coupled to the second robotic arm 10534. The first scope device 10512 is inserted into the abdominal cavity 10505 (e.g., extraluminal anatomical space) through the first trocar 10536. The first trocar 10536 is coupled to the robotic arm 10538. The first scope device 10512 is further inserted into the colonic lumen 10507 (e.g., intraluminal anatomical space) through the lumen of the sealing port 10540, with the sealing port 10540 positioned within the wall 10506 of the colon 10502. The first scope device 10512 can be inserted into and through the first trocar 10536 and sealing port 10540 such that the first portion 10512a of the first scope device 10512 is located in the abdominal cavity 10505 (e.g., extraluminal anatomical space) and the second portion 10512b of the first scope device 10512, distal to the first portion 10512a, is positioned in the colonic lumen 10507 (e.g., intraluminal anatomical space). In some embodiments, the sealing port 10540 is configured to prevent the contents of the colonic lumen 10507 from escaping into the abdominal cavity 10505 during the ventilation procedure. In other embodiments, the sealing port 10540 can be omitted so that the first scope device 10512 is inserted directly through an incision formed in the colonic wall 10502a.

[0198] As shown, the first scope device 10512 has a first portion 10512a located in the abdominal cavity 10505 and a second portion 10512b located distal to the first portion 10512a and located in the colonic cavity 10507. That is, the first scope device 10512 is designed to be introduced into the colon 10502 by a laparoscopic approach. Before or after insertion of the second portion 10512b of the first scope device 10512, sealing clips 10509a and 10509b can be positioned on the opposing ends of the ventilated region of the colon 10502.

[0199] The first scope device 10512 includes a first vent port 10523 configured to vent the colon lumen 10507 prior to or simultaneously with the insertion of any device or instrument into the colon lumen 10507. In this illustrated embodiment, the first vent port 10523 is in fluid communication with a second working channel 10526 of the first scope device 10512. As a result, the first vent port 10523 can be used to control the inflow of fluid into or outflow of fluid from the colon lumen 10507, thereby venting or deflating the colon lumen 10507. Although not shown, the first vent port 10523 is connected to a fluid system. The fluid system may include a pump and a fluid reservoir. The pump generates pressure to push fluid into the colonic lumen 10507, causing it to expand (e.g., pressurize), and generates suction to draw fluid out of the colonic lumen 10507 to contract (e.g., depressurize) the colonic lumen 10507. The fluid entering and leaving the colonic lumen 10507 can be any suitable fluid (e.g., saline solution, carbon dioxide gas, etc.). In other embodiments, the colonic lumen 10507 can be ventilated and ventilated using any other suitable ventilation mechanisms and devices.

[0200] In some embodiments, the first portion 10512a of the first scope device 10512 may be driven from one or more tool drivers (not shown) located in a motor housing 10521, positioned between the robot arm 10538 and the first trocar 10536.

[0201] As further shown in Figure 27, once the second portion 10512b of the first scope device 10512 is positioned within the colonic lumen 10507, the first instrument 10532 can be inserted through the first working channel 10524 of the first scope device 10512 so that the distal end of the first instrument is positioned within the colonic lumen 10507. As a result, the first instrument 10532 is present in both the abdominal cavity (e.g., extraluminal anatomical space) and the colonic lumen 10507 (e.g., intraluminal anatomical space). Once inserted, the end effector 10533 (of the first instrument 10532) can interact with the tumor 10501 for subsequent removal. The end effector 10533 can have various configurations, but in this illustrated embodiment, the end effector 10533 is in the form of a pair of movable jaws. In some embodiments, at least one of the first working channel 10524 and the second working channel 10526 is configured to allow instrument replacement without impairing the position of the first scope device 10512 in at least one of the abdominal cavity 10505 and the colon cavity 10507. This also allows for maintaining the field of view of the first imaging system 10528.

[0202] In some embodiments, the first scope device 10512 can be configured to form a seal within the colonic lumen 10507. For example, as shown in Figure 27, the first scope device 10512 includes a sealing element 10531 positioned at or near the distal end of the first scope device 10512. The sealing element 10531 can have various configurations, but in this illustrated embodiment, the sealing element 10531 is in the form of an inflatable annular ring positioned around the first scope device 10512. While the first scope device 10512 is advanced through the abdominal cavity 10505 into the colonic lumen 10507, the sealing element 10531 is in a contracted state. Once inside the colonic lumen 10507, the sealing element 10531 can be inflated to form a seal by engaging with the internal tissue wall of the colon 10502. Furthermore, in certain embodiments, the sealing element 10531 can also function as a fixed point for the first scope device 10512 within the colonic lumen 10507 when in an expanded state.

[0203] As further shown in Figure 27, the second scope device 10514 is positioned laparoscopically within the abdominal cavity 10505. The second scope device includes a flexible body 10552 through which third and fourth working channels 10554, 10556 extend, and a second imaging system 10558 (e.g., a camera) at its distal end. The flexible body 10552 can be formed from any suitable flexible material(s).

[0204] The second scope device 10514 is inserted into the abdominal cavity 10505 (e.g., extraluminal anatomical space) through a second trocar 10566 positioned within the abdominal wall 10503. The second trocar 10566 is coupled to a second robotic arm 10568. The second scope device 10514 is inserted and positioned within the abdominal cavity 10505 (e.g., extraluminal anatomical space). The second trocar 10566 includes a second ventilation port 10567 configured to ventilate the abdominal cavity 10105 prior to or simultaneously with the insertion of any device or instrument into the abdominal cavity 10105. In this illustrated embodiment, the second ventilation port 10567 is in fluid communication with a fourth working channel 10556. As a result, the second vent port 10567 can be used to control the inflow or outflow of fluid into or out of the abdominal cavity 10505 in order to ventilate or deflate the abdominal cavity 10505. Although not shown, the second vent port 10567 is connected to a fluid system. The fluid system may include a pump and a fluid reservoir. The pump generates pressure to push fluid into and expand (e.g., pressurize) the abdominal cavity 10105, and generates suction to draw fluid out of the abdominal cavity 10105 in order to contract (e.g., depressurize) the abdominal cavity 10105. The fluid entering and leaving the abdominal cavity 10105 may be any suitable fluid (e.g., saline solution, carbon dioxide gas, etc.). In other embodiments, the abdominal cavity 10105 may be ventilated and deflated using any other suitable ventilating mechanisms and devices.

[0205] In some embodiments, the flexible body 10552 of the second scope device 10514 can be driven from one or more tool drivers (not shown) located in a motor housing 10551, positioned between the second robot arm 10568 and the second trocar 10566.

[0206] As shown, during use, the proximal end of the second scope device 10514 is coupled to the first robotic arm 10560, and the second instrument 10562 is coupled to the second robotic arm 10564. The second instrument 10562 extends into and through the third working channel 10554 into the abdominal cavity 10505 (e.g., extraluminal anatomical space). The second instrument 10562 can have various configurations, but in this illustrated embodiment, the second instrument 10562 has an elongated shaft 10562a with an end effector 10563 at its distal end. In some embodiments, the second instrument 10562 is configured to assist in manipulating the colon 10502 from the abdominal cavity 10505 (e.g., extraluminal anatomical space) in order to position the first instrument 10532 within the colonic lumen 10507 (e.g., intraluminal anatomical space). Furthermore, the end effector 10563 of the second instrument 10562 can interact with the colon 10502 to help stabilize the colon 10502 for the removal of the tumor 10501. The end effector 10563 can have various configurations, but in this illustrated embodiment, the end effector 10563 is in the form of a pair of movable jaws. In some embodiments, the end effector 10563 can be used to form a seal within the colonic lumen 10507 (for example, by clamping the colon 10502).

[0207] During use, the colonic cavity 10507 is pressurized to a first pressure via fluid entry through the first ventilation port 10523 and the second working channel 10526. In addition, the abdominal cavity 10505 is pressurized to a second pressure via fluid entry through the second ventilation port 10567 and the fourth working channel 10556. In some embodiments, the first pressure is different from the second pressure, or the first and second pressures are the same.

[0208] The first and second pressures can be adjusted independently to alter the working volume space within the abdominal cavity 10505, the colonic cavity 10507, or both. For example, the working volume space within the abdominal cavity 10505 can be increased by increasing the pressure within the abdominal cavity 10505, decreasing the pressure within the colonic cavity 10507, or both. Similarly, the working volume space within the abdominal cavity 10505 can be decreased by decreasing the pressure within the abdominal cavity 10505, increasing the pressure within the colonic cavity 10507, or both. Furthermore, the working volume space within the colonic cavity 10507 can be increased by increasing the pressure within the colonic cavity 10507, or decreased by decreasing the pressure within the colonic cavity 10507.

[0209] Although not shown, the first imaging system 10528 and the second imaging system 10558 are connected to one or more displays that provide snapshots and / or live video feeds of the surgical site. The snapshots and / or live video feeds on the displays can enable the physician to observe the surgical site from multiple angles and approaches, for example. As a result, the first imaging system 10528 and the second imaging system 10558 can provide the physician with information that can be used to determine the effective working volume space for the first and second instruments and, if present, any adjustments that need to be made to the first ventilation space, the second ventilation space, or both, for a particular surgical task or step, or throughout the entire surgical procedure.

[0210] The surgical systems described herein may be designed to be discarded after a single use, or they may be designed to be used multiple times. However, in either case, the surgical system may be readjusted for reuse after at least one use. Readjustment may include any combination of the steps of disassembling the surgical device, followed by cleaning or replacing specific parts, and then reassembling. In particular, the surgical system may be disassembled, and any number of specific pieces or parts of the surgical system may be selectively replaced or removed in any combination. After cleaning and / or replacing specific parts, the surgical system may be reassembled for later use in a readjustment facility or by the surgical team immediately before the surgical procedure. Those skilled in the art will understand that various techniques for disassembly, cleaning / replacement, and reassembly are available for readjusting surgical devices. The use of such techniques and the resulting readjusted instruments are all within the scope of this application.

[0211] Furthermore, in this disclosure, components with similar names in embodiments generally have similar characteristics, and therefore, in a particular embodiment, each characteristic of each component with a similar name is not necessarily described in full detail. In addition, to the extent that linear or circular dimensions are used in the description of the disclosed systems, devices, and methods, such dimensions are not intended to limit the types of shapes that can be used in combination with such systems, devices, and methods. Those skilled in the art will recognize that dimensions equivalent to such linear and circular dimensions can be readily determined for any geometric shape. The size and shape of systems and devices, and their components, may depend at least on the anatomical structure in which the systems and devices are used, the size and shape of the components in which the systems and devices are used together, and the methods and surgeries in which the systems and devices are used.

[0212] It will be recognized herein that the terms “proximal” and “distal” are used relative to the user, such as a clinician holding the instrument's handle. It will be understood herein that the terms “proximal” and “distal” are used with respect to the upper end (e.g., the end furthest from the surgical site during use) and lower end (e.g., the end closest to the surgical site during use) of a surgical instrument configured to be attached to a robot, respectively. Other spatial terms, such as “anterior” and “posterior,” similarly correspond to distal and proximal, respectively. For convenience and clarity, it will also be understood herein that spatial terms such as “vertical” and “horizontal” are used relative to drawings. However, surgical instruments are used in many orientations and positions, and these spatial terms are not intended to be limited or absolute.

[0213] Values ​​or ranges may be expressed herein as “approximately” and / or “approximately” from one specific value to another specific value. Where values ​​or ranges are expressed in this manner, other embodiments disclosed include the enumerated specific values ​​and / or from one specific value to another specific value. Similarly, where values ​​are expressed in an approximate form by the preceding use of “approximately,” it will be understood that many values ​​disclosed herein are enumerated and that different embodiments are formed by their specific values. It will also be understood that many values ​​are disclosed herein, and each value is disclosed herein as a value that is “approximately” in addition to its specific value itself. In each embodiment, “approximately” may be used to mean, for example, within 10% of the enumerated values, within 5% of the enumerated values, or within 2% of the enumerated values.

[0214] For the purpose of explaining and defining these instructions, unless otherwise stated, the term “substantially” is used herein to describe the degree of inherent uncertainty that may arise from any quantitative comparison, value, measurement, or other expression. The term “substantially” is also used herein to describe the extent to which a quantitative expression may deviate from the described standard without altering the fundamental function of the object in question.

[0215] Those skilled in the art will recognize further features and advantages of the present invention based on the embodiments described above. Therefore, the present invention is not limited to what is specifically shown and described, except as provided by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety. Any patent, publication or information incorporated herein by reference in whole or in part is limited to the extent that the incorporated elements do not contradict existing definitions, descriptions, or elements of other disclosures contained herein. Therefore, the disclosures expressly provided herein take precedence over any contradictory elements incorporated herein by reference.

[0216] [Implementation Method] (1) A surgical system, A first scope device having a first part configured to be inserted and positioned in an extraluminal anatomical space, and a second part located distal to the first part and configured to be positioned in an intraluminal anatomical space, wherein the first scope device is The device comprises a flexible body having a working channel extending through its interior, and a first imaging system at the distal end of the first scope device, wherein the working channel is configured to allow the distal end of the first instrument to be inserted into the extraluminal anatomical space and then inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal space and the intraluminal space. The first scope device, A surgical system comprising: a second instrument configured to be inserted into the extraluminal anatomical space, and configured to be coupled to the first part of the first scope device within the extraluminal anatomical space to facilitate the movement of the second part of the first scope device while the second part is positioned within the intraluminal anatomical space; and a second instrument configured to move the first part. (2) The surgical system according to Embodiment 1, wherein the second instrument is located in the extraluminal anatomical space and is configured to be coupled to the first portion of the first scope device at a predetermined position directly adjacent to a tissue wall defining at least a portion of the intraluminal anatomical space. (3) The surgical system according to Embodiment 1 or Embodiment 2, wherein the second instrument comprises a rigid shaft having an end effector at its distal end, and the end effector is configured to be coupled to the first part of the first scope device. (4) The surgical system according to Embodiment 1, further comprising a cannula having a lumen extending through the interior, wherein the cannula is positioned within a tissue wall defining at least a portion of the intraluminal anatomical space, and the distal end of the flexible body is configured to allow insertion from the extraluminal anatomical space through the lumen into the intraluminal anatomical space. (5) The surgical system according to Embodiment 4, wherein the second instrument is further configured to be coupled to the cannula and move the cannula in order to facilitate the movement of the second portion of the first scope device while the distal end of the flexible body is in the intraluminal anatomical space.

[0217] (6) The surgical system according to Embodiment 5, wherein the second instrument comprises a rigid shaft having an end effector at its distal end, and the end effector is configured to be coupled to the cannula. (7) The surgical system according to Embodiment 1, further comprising a fluid port configured to vent to the extraluminal anatomical space. (8) A surgical system, An anchor member positioned within the extraluminal anatomical space and configured to contact a tissue wall that at least partially defines the intraluminal anatomical space, A cannula having a first portion configured to be inserted and positioned in the extraluminal anatomical space, and a second portion located distal to the first portion and configured to be positioned in the intraluminal anatomical space, wherein the cannula is configured such that the distal end of the first instrument is inserted into the extraluminal anatomical space and can be inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal anatomical space and the intraluminal anatomical space. A surgical system comprising: a selectively deployable stabilizing member positioned on the first portion of the cannula in the extraluminal anatomical space, configured to connect to the anchor member when deployed to provide an anchor point for the first instrument, thereby facilitating the rotational movement of the first instrument within the intraluminal anatomical space. (9) The surgical system according to embodiment 8, wherein the anchor member is further configured to seal a portion of the intraluminal anatomical space. (10) The surgical system according to Embodiment 8 or Embodiment 9, further comprising a magnet disposed within the anchor member, wherein the magnet is configured to connect the selectively deployable stabilizing member to the anchor member when the selectively deployable stabilizing member is in a deployed state.

[0218] (11) The surgical system according to Embodiment 8, further comprising a first scope device, the first scope device being inserted into and through the lumen of the cannula such that a first portion of the scope device is located in the extraluminal anatomical space and a second distal portion of the first portion is located in the intraluminal anatomical space.

Claims

1. It is a surgical system, A first scope device having a first portion configured to be inserted and positioned in an extraluminal anatomical space, and a second portion located distal to the first portion and configured to be positioned in an intraluminal anatomical space, wherein the first scope device is The device comprises a flexible body having a working channel extending through its interior, and a first imaging system at the distal end of the first scope device, wherein the working channel is configured to allow the distal end of the first instrument to be inserted into the extraluminal anatomical space and then inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal space and the intraluminal space. The first scope device, A surgical system comprising: a second instrument configured to be inserted into the extraluminal anatomical space, and configured to be coupled to the first part of the first scope device within the extraluminal anatomical space to facilitate the movement of the second part of the first scope device while the second part is positioned within the intraluminal anatomical space; and the second instrument configured to move the first part.

2. The surgical system according to claim 1, wherein the second instrument is located in the extraluminal anatomical space and is configured to be coupled to the first portion of the first scope device at a predetermined position directly adjacent to a tissue wall defining at least a portion of the intraluminal anatomical space.

3. The surgical system according to claim 1 or 2, wherein the second instrument comprises a rigid shaft having an end effector at its distal end, and the end effector is configured to be coupled to the first portion of the first scope device.

4. The surgical system according to claim 1, further comprising a cannula having a lumen extending through the interior, wherein the cannula is configured to be positioned within a tissue wall defining at least a portion of the intraluminal anatomical space, and the distal end of the flexible body is configured to allow insertion from the extraluminal anatomical space through the lumen into the intraluminal anatomical space.

5. The surgical system according to claim 4, wherein the second instrument is further configured to be coupled to the cannula and move the cannula in order to facilitate the movement of the second portion of the first scope device while the distal end of the flexible body is in the intraluminal anatomical space.

6. The surgical system according to claim 5, wherein the second instrument comprises a rigid shaft having an end effector at its distal end, and the end effector is configured to be coupled to the cannula.

7. The surgical system according to claim 1, further comprising a fluid port configured to vent to the extraluminal anatomical space.

8. It is a surgical system, An anchor member positioned within the extraluminal anatomical space and configured to contact a tissue wall that at least partially defines the intraluminal anatomical space, A cannula having a first portion configured to be inserted and positioned in the extraluminal anatomical space, and a second portion located distal to the first portion and configured to be positioned in the intraluminal anatomical space, wherein the cannula is configured such that the distal end of the first instrument is inserted into the extraluminal anatomical space and can be inserted through the extraluminal anatomical space into the intraluminal anatomical space, so that the first instrument is present in both the extraluminal anatomical space and the intraluminal anatomical space. A surgical system comprising: a selectively deployable stabilizing member positioned on the first portion of the cannula in the extraluminal anatomical space, configured to connect to the anchor member when deployed to provide an anchor point for the first instrument, thereby facilitating the rotational movement of the first instrument within the intraluminal anatomical space.

9. The surgical system according to claim 8, wherein the anchor member is further configured to seal a portion of the intraluminal anatomical space.

10. The surgical system according to claim 8 or 9, further comprising a magnet disposed within the anchor member, wherein the magnet is configured to connect the selectively deployable stabilizing member to the anchor member when the selectively deployable stabilizing member is in a deployed state.

11. The surgical system according to claim 8, further comprising a first scope device, the first scope device being configured to be inserted into and through the lumen of the cannula such that a first portion of the scope device is located in the extraluminal anatomical space and a second distal portion of the first portion is located in the intraluminal anatomical space.

Images