Method and system for controlling coordinated surgical instruments

Abstract

Systems, devices, and methods are provided for controlling coordinated surgical instruments. Various aspects of the present disclosure provide for coordinated operation of surgical instruments that access a common body cavity of a patient from different approaches to achieve a common surgical goal. For example, the various methods, devices, and systems disclosed herein can enable coordinated treatment of surgical tissue with heterogeneous minimally invasive surgical systems that approach tissue from various anatomical spaces and operate in concert with one another to provide a desired surgical treatment.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 249,870, filed on 29 September 2021, entitled “Methods and Systems for Controlling Cooperative Surgical Instruments,” the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] Some surgical procedures require the use of multiple surgical instruments acting simultaneously on a region or portion of tissue in order to successfully perform the procedure. In some situations, due to anatomical limitations and / or the nature of the procedure, it is impossible for multiple surgical instruments to directly visually contact each other, even if they can be located within the same anatomical space. For example, during a procedure acting on a shared tissue structure (e.g., a section of a patient's small intestine), multiple surgical instruments may need to be positioned on visually separated parts of the shared tissue structure in order to successfully perform the procedure.

[0003] However, in some implementations, a first surgical instrument and a second surgical instrument, both operating on a tissue area, may operate through independent systems, even if the instruments share a common surgical objective. In such scenarios, neither instrument can directly visualize the movement of the other, but in situations where coordinated operation of the first and second surgical instruments is required to successfully perform the procedure, it may be difficult or impossible for the first and second surgical instruments to operate in coordination to achieve a shared successful surgical outcome. [Overview of the project] [Problems that the invention aims to solve]

[0004] Accordingly, there remains a need for improved methods and systems for controlling cooperating surgical instruments when direct visualization between the cooperating surgical instruments is restricted, for example, by surrounding tissue. [Means for Solving the Problems]

[0005] In one aspect, there is provided a system comprising a first surgical instrument inserted into a first portion of a body cavity and configured to operate on a first surgical treatment site located within the body cavity of a patient. A second surgical instrument inserted into a second portion of the body cavity and configured to operate on a second surgical treatment site located within the body cavity is also provided. The second portion of the body cavity is different from the first portion of the body cavity, and the second surgical treatment site is different from the first treatment tissue site. Further, the system comprises a first flexible endoscope having a first image sensor configured such that the second surgical instrument is positioned in the first portion of the body cavity so as not to be within the field of view of the first image sensor. A second flexible endoscope configured such that the first surgical instrument is positioned in the second portion of the body cavity so as not to be within the field of view of the second image sensor is also provided. Additionally, the system is configured to receive images collected by each of the first image sensor and the second image sensor, determine a first position of the first surgical instrument and a second position of the second surgical instrument, determine a distance and an orientation of the first surgical instrument relative to the second surgical instrument, and cause movement of at least one of the first surgical instrument and the second surgical instrument within the body cavity based on the determined distance and orientation. comprising a controller configured to

[0006] The system can have a number of variations. For example, the first surgical treatment site can be adjacent to a first proximal anatomical landmark, the second surgical treatment site can be adjacent to a second distal anatomical landmark, and the first surgical treatment site and the second surgical treatment site can be spaced apart from each other within the body cavity. In yet another embodiment, the first proximal anatomical landmark may be the duodenojejunal flexure, and the second distal anatomical landmark may be the ileocecal valve.

[0007] In some embodiments, a first surgical instrument may be configured to be inserted into a body cavity through a first natural orifice of the patient, and a second surgical instrument may be configured to be inserted into a body cavity through a second, different natural orifice of the patient. In other embodiments, the controller may control the speed of movement of at least one of the first and second surgical instruments within the body cavity, based at least on a determined location and distance. In yet another embodiment, the system may include a first portion of a surgical implant that is releasably attached to the first surgical instrument and configured to be delivered into the body cavity while releasably attached to the first surgical instrument, and a second portion of the surgical implant that is releasably attached to the second surgical instrument and configured to be delivered into the body cavity while releasably attached to the second surgical instrument. In some embodiments, the controller may be configured to trigger movement of at least one of the first and second surgical instruments before the first and second portions of the surgical implant are delivered into the body cavity. In other embodiments, after the first and second portions of the implant have been delivered into the body cavity, the controller may be configured to move a first surgical instrument within the body cavity to move the first portion of the surgical implant relative to the second portion of the surgical implant, and to move a second surgical instrument within the body cavity to move the second portion of the surgical implant relative to the first portion of the surgical implant. In some embodiments, the first portion of the surgical implant may include a first electromagnetic tracker configured to provide data relating to the first portion of the implant to the controller, and the second portion of the surgical implant may include a second electromagnetic tracker configured to provide data relating to the second portion of the implant to the controller. In some embodiments, at least one of the movements of the first and second surgical instruments may be based on the received data relating to the first and second portions of the implant. In some embodiments, the body cavity may include the jejunum, and the surgical implant may include an anastomotic device.

[0008] In another aspect, there is provided a system comprising at least one data processor and a memory storing instructions configured to cause the at least one data processor to perform operations. The operations include receiving, in real time, first image data characterizing a first portion of a patient's body cavity from a first image sensor of a first flexible endoscopic system. The operations also include receiving, in real time, second image data characterizing a second portion of the body cavity from a second image sensor of a second flexible endoscopic system, wherein the second portion of the body cavity is different from the first portion of the body cavity. The operations further include determining a first position of a first surgical instrument based on the first image data, and determining a second position of a second surgical instrument relative to the first surgical instrument based on the second image data. The operations also include controlling an advancement speed and an advancement force of the first surgical instrument and the second surgical instrument, wherein the advancement speed and the advancement force are limited by the detected proximity and orientation of the respective distal ends of the first surgical instrument and the second surgical instrument relative to each other.

[0009] The system can have several different variations. For example, a first surgical treatment site can be adjacent to a first proximal anatomical landmark, a second surgical treatment site can be adjacent to a second distal anatomical landmark, and the first surgical treatment site and the second surgical treatment site can be spaced apart from each other within the body cavity. In yet another example, the first proximal anatomical landmark can be the duodenojejunal flexure, and the second distal anatomical landmark can be the ileocecal valve.

[0010] In some embodiments, a first surgical instrument may be configured to be inserted into a body cavity through a first natural orifice of the patient, and a second surgical instrument may be configured to be inserted into a body cavity through a second, different natural orifice of the patient. In one embodiment, the operation of at least one data processor further includes deploying a first portion of a surgical implant that is releasably attached to the first surgical instrument and configured to be delivered into the body cavity while releasably attached to the first surgical instrument, and deploying a second portion of the surgical implant that is releasably attached to the second surgical instrument and configured to be delivered into the body cavity while releasably attached to the second surgical instrument. In another embodiment, the body cavity includes the jejunum, and the surgical implant includes an anastomotic device.

[0011] In another embodiment, a method is provided which includes receiving first image data in real time from a first image sensor of a first endoscope system, characterizing a first portion of a patient's body cavity. The method also includes receiving second image data in real time from a second image sensor of a second endoscope system, characterizing a second portion of the body cavity. The method further includes determining, based on the first image data, a first location of a first surgical instrument, which is located within the first portion of the patient's body cavity and configured to act on a first surgical treatment site within the body cavity, and the first surgical instrument is outside the field of view of the second endoscope system. The method also includes determining, based on the second image data, a second location of a second surgical instrument relative to the first surgical instrument. The second surgical instrument is located within the second portion of the body cavity and configured to act on a second surgical treatment site within the body cavity, and the second surgical instrument is also outside the field of view of the first endoscope system. Additionally, the method includes determining the distance and orientation of the first surgical instrument relative to the second surgical instrument, and, based on the determined distance and orientation, causing movement of at least one of the first and second surgical instruments within the body cavity.

[0012] The method can have numerous variations. In one embodiment, the method further includes advancing a first surgical instrument into a body cavity through a first natural orifice of the patient, and advancing a second surgical instrument into a body cavity through a second different natural orifice of the patient. In another embodiment, the method includes determining the orientation of a first portion and a second portion of a surgical implant that are releasably engaged with the first and second surgical instruments, respectively. In yet another embodiment, the method includes controlling the speed of movement of at least one of the first and second surgical instruments into the body cavity, based at least on the determined location and distance.

[0013] In another embodiment, a system is provided comprising a first surgical instrument and a second surgical instrument, and a first flexible endoscope and a second flexible endoscope. The first surgical instrument is inserted into a first portion of a body cavity and configured to act on a first surgical treatment site located within the patient's body cavity, and the second surgical instrument is inserted into a second portion of a body cavity and configured to act on a second surgical treatment site located within the body cavity. Additionally, the second portion of the body cavity is distinct from the first portion of the body cavity, and the second surgical treatment site is distinct from the first treatment tissue site. Furthermore, the first flexible endoscope has a first image sensor and is configured to be positioned so that the second surgical instrument is not within the field of view of the first image sensor, and the second flexible endoscope has a second image sensor and is configured so that the first surgical instrument is not within the field of view of the second image sensor. The system also has a controller configured to receive images collected by the first and second image sensors, determine the first location of the first surgical instrument and the second location of the second surgical instrument relative to each other, and trigger synchronized surgical actions between the first and second surgical instruments at the first and second treatment tissue sites, respectively.

[0014] The system can have a number of different variants. For example, the system may further include a first portion of a surgical implant configured to be releasably attached to a first surgical instrument and delivered into a body cavity while being releasably attached to the first surgical instrument, and a second portion of a surgical implant configured to be releasably attached to a second surgical instrument and delivered into a body cavity while being releasably attached to the second surgical instrument. In some embodiments, the controller may also be configured to actuate the deployment of the first and second portions of the surgical implant simultaneously. In another embodiment, the body cavity may include the jejunum, and the surgical implant may include an anastomotic device. In yet another embodiment, the first portion of the surgical implant may include a first electromagnetic tracker configured to provide data about the first portion of the implant to the controller, and the second portion of the surgical implant may include a second electromagnetic tracker configured to provide data about the second portion of the implant to the controller. In some embodiments, the simultaneous deployment of the first and second portions by the controller may be based on received data about the first and second portions of the implant.

[0015] In another embodiment, the system may further include a third surgical instrument configured to be introduced into a third portion of the body cavity and also configured to assist the controller in causing synchronized surgical action of the first and second surgical instruments. In yet another embodiment, the first surgical instrument may be configured to be introduced into the patient through a first natural orifice of the patient, the second surgical instrument may be configured to be introduced into the patient through a second different natural orifice of the patient, and the third surgical instrument may be configured to be introduced into the patient from a laparoscopic approach. In yet another embodiment, the synchronized surgical action between the first and second surgical instruments may include simultaneous synchronized surgical action at a first and second treatment tissue site.

[0016] In some embodiments, the controller can be configured to trigger synchronized action between the first and second surgical instruments when tissue obstructs the field of view of the first endoscope and when tissue obstructs the field of view of the second endoscope.

[0017] In another embodiment, a system is provided that includes at least one data processor and a memory for storing instructions configured to cause the at least one data processor to perform an operation. The operation includes receiving first image data in real time from a first image sensor of a first endoscope, characterizing a first portion of a patient's body cavity. The operation also includes receiving second image data in real time from a second image sensor of a second endoscope, characterizing a second portion of the body cavity. The operation further includes determining, based on the first image data, a first location of a first surgical instrument configured to act on tissue at a first surgical treatment site within the first portion of the body cavity. Furthermore, the first surgical instrument is outside the field of view of the second endoscope. The operation also includes determining, based on the second image data, a second location of a second surgical instrument relative to the first location of the first surgical instrument. The second surgical instrument is configured to act on tissue at a second surgical treatment site, and the second surgical instrument is outside the field of view of the first endoscope. The operation also includes causing synchronized surgical action between the first surgical instrument and the second surgical instrument in the first and second treatment tissue sites, respectively.

[0018] The system can have numerous different variants. In one embodiment, the synchronized surgical action may include simultaneously deploying a first portion of a surgical implant from a first surgical instrument and a second portion of the surgical implant from a second surgical instrument. In yet another embodiment, the body cavity includes the jejunum, and the surgical implant includes a two-part magnetic anastomosis device. In yet another embodiment, the system may include receiving third image data characterizing a third portion of the patient's body cavity in real time from a third image sensor of a third endoscope. In some embodiments, the synchronized surgical action may include avoiding penetration into any tissue by the first and second surgical instruments.

[0019] In yet another embodiment, a method is provided which includes receiving first image data in real time from a first image sensor of a first endoscope system, characterizing a first portion of a patient's body cavity. The method also includes receiving second image data in real time from a second image sensor of a second endoscope system, characterizing a second portion of the body cavity. The method also includes, by means of a controller, determining a first location of a first surgical instrument for manipulating tissue at a first surgical treatment site located within the first portion of the patient's body cavity, based on the first image data, wherein the first surgical instrument is outside the field of view of the second endoscope system. The method further includes, by means of a controller, determining a second location of a second surgical instrument relative to the first surgical instrument, based on the second image data. The second surgical instrument manipulates tissue at a second surgical treatment site located within the second portion of the body cavity, and the second surgical instrument is outside the field of view of the first endoscope system. This method further includes using a controller to trigger synchronized surgical actions between a first surgical instrument and a second surgical instrument at a first and second treatment tissue site, respectively.

[0020] The method can have a number of different variations. For example, the method may further include deploying a first portion of a surgical implant that is releasably attached to a first surgical instrument and configured to be delivered into a body cavity while releasably attached to the first surgical instrument, and deploying a second portion of the surgical implant that is releasably attached to a second surgical instrument and configured to be delivered into a body cavity while releasably attached to the second surgical instrument. In another embodiment, the body cavity includes the jejunum, and the surgical implant includes a two-part magnetic anastomosis device. In yet another embodiment, the method further includes receiving third image data characterizing a third portion of the patient's body cavity in real time from a third image sensor of a third endoscope.

[0021] In another embodiment, a system is provided which includes a first surgical instrument configured to be inserted into a first body cavity and to deploy a first portion of a surgical implant within the patient's body cavity. The system also includes a second surgical instrument configured to be inserted into a second body cavity and to deploy a second portion of a surgical implant within the body cavity, the second body cavity being distinct from the first. The system further includes a first flexible endoscope having a first image sensor, the first flexible endoscope being positioned so that the second surgical instrument is not within the field of view of the first image sensor. The system also includes a second flexible endoscope with a second image sensor, the second flexible endoscope being positioned so that the first surgical instrument is not within the field of view of the second image sensor. The system also includes a controller configured to receive images collected by the first and second image sensors, determine the first location of the first surgical instrument and the second location of the second surgical instrument relative to each other, determine the characteristics of the tissue walls within the first and second portions of the first body cavity, and determine the placement locations of the first and second portions of a surgical implant based on the characteristics of the tissue walls.

[0022] The system can have several variations. For example, a first part of a surgical implant may include a first electromagnetic tracker configured to provide data relating to the first part of the implant to the controller, and a second part of the surgical implant may include a second electromagnetic tracker configured to provide data relating to the second part of the implant to the controller. In some embodiments, the determined placement locations of the first and second parts of the surgical implant may be based on at least received data relating to the first and second parts of the implant. In another embodiment, the tissue wall properties may include at least one of thickness, stiffness, or tissue composition. In yet another embodiment, the controller may be configured to determine the thickness of the tissue wall based on at least the first and second locations of the first and second instruments. In yet another embodiment, the controller may be configured to determine the tissue wall properties based on at least one of tissue impedance and non-visible light spectral imaging.

[0023] In some embodiments, the controller may be configured to determine the locations of the first and second surgical instruments when tissue obstructs the view of the first endoscope and when tissue obstructs the view of the second endoscope. In some embodiments, the first surgical instrument may be configured to be inserted into the body cavity through a first natural orifice of the patient, and the second surgical instrument may be configured to be inserted into the body cavity through a second different natural orifice of the patient. In other embodiments, the controller may be configured to rotate and articulate the first surgical instrument to position the first portion of the surgical implant relative to the second portion of the surgical implant. In yet another embodiment, the body cavity may include the jejunum, and the surgical implant may include a two-part magnetic anastomosis device.

[0024] In another embodiment, a system is provided having at least one data processor and a memory for storing instructions configured to cause the at least one data processor to perform an operation. The operation includes receiving first image data in real time from a first image sensor of a first endoscope, characterizing a first portion of a patient's body cavity. The operation also includes receiving second image data in real time from a second image sensor of a second endoscope, characterizing a second portion of the first body cavity. Furthermore, the operation includes determining a first location of a first surgical instrument configured to deploy a first portion of a surgical implant within the first portion of the body cavity, based on the first image data. The operation also includes determining a second location of a second surgical instrument relative to the first location of the first surgical instrument, based on the second image data, the second surgical instrument configured to deploy a second portion of a surgical implant within the second portion of the body cavity. The operation also includes determining the characteristics of the tissue walls within the first and second portions of the first body cavity, and determining the placement locations of the first and second portions of the surgical implant, based on the characteristics of the tissue walls.

[0025] The system can have several different variants. For example, the operation of at least one data processor may include receiving data relating to a first portion of the implant from a first electromagnetic tracker in the first portion of the surgical implant to the controller, and receiving data relating to a second portion of the implant from a second electromagnetic tracker in the second portion of the surgical implant to the controller. In some embodiments, the operation may also include determining the placement locations of the first and second portions of the surgical implant based at least on the data received from the first and second electromagnetic trackers. In another embodiment, the tissue wall properties may include at least one of thickness, stiffness, or tissue composition. In yet another embodiment, the system may determine the tissue wall properties based on at least one of the first and second locations of the first and second instruments, tissue impedance, and non-visible light spectral imaging. In yet another embodiment, the system may include determining a first location of the first surgical instrument and a second location of the second surgical instrument when tissue obstructs the field of view of the first endoscope and when tissue obstructs the field of view of the second endoscope. In yet another embodiment, the body cavity may include the jejunum, and the surgical implant may include an anastomotic device.

[0026] In yet another embodiment, a method is provided which includes receiving first image data in real time from a first image sensor of a first endoscope system, characterizing a first portion of a patient's body cavity. The method also includes receiving second image data in real time from a second image sensor of a second endoscope system, characterizing a second portion of a first hollow organ. The method also includes, by means of a controller, determining a first location of a first surgical instrument located within the first body portion and with a first portion of a surgical implant releasably engaged. The first surgical instrument is outside the field of view of the second endoscope system. Furthermore, the second portion of the body cavity is different from the first portion, and the second surgical treatment site of the body cavity is different from the first surgical treatment site. The method further includes, by means of a controller, determining a second location of a second surgical instrument in the second portion of the body cavity relative to the first surgical instrument, based on the second image data. Additionally, the second surgical instrument has a second portion of a surgical implant that is releasably engaged with the second surgical instrument, and the second surgical instrument is outside the field of view of the first endoscope system. The method further includes determining the characteristics of the tissue walls in the first and second portions of the first body cavity by a controller, and determining the placement of the first and second portions of the surgical implant by the controller based on the characteristics of the tissue walls.

[0027] The method can have a number of different variations. For example, the properties of the tissue wall can include at least one of thickness, rigidity, or tissue composition. In another embodiment, the method can include determining the properties of the tissue wall based on at least one of first and second locations of a first and second instrument, tissue impedance, and non-visible light spectral imaging. In yet another embodiment, the method can also include determining the first location of the first surgical instrument and the second location of the second surgical instrument when the tissue obstructs the field of view of the first endoscope and when the tissue obstructs the field of view of the second endoscope. In yet another embodiment, the body cavity can include the jejunum, and the surgical implant can include an anastomotic device. [Brief explanation of the drawing]

[0028] The present invention will be described with reference to the accompanying drawings. [Figure 1] This is a schematic diagram of one embodiment of a surgical visualization system. [Figure 2] Figure 1 is a schematic diagram of the triangulation between the surgical device, imaging device, and critical structure. [Figure 3] This is a schematic diagram of another embodiment of the surgical visualization system. [Figure 4] This is a schematic diagram of one embodiment of a control system for a surgical visualization system. [Figure 5] This is a schematic diagram of one embodiment of the control circuit of a control system for a surgical visualization system. [Figure 6] This is a schematic diagram of one embodiment of a combination logic circuit for a surgical visualization system. [Figure 7] This is a schematic diagram of one embodiment of the sequential logic circuit of a surgical visualization system. [Figure 8] This is a perspective view of yet another embodiment of the surgical visualization system. [Figure 9] This is a schematic diagram of another embodiment of a control system for a surgical visualization system. [Figure 10] This graph shows the wavelength versus absorption coefficient for various biological substances. [Figure 11] This is a schematic diagram of one embodiment of a spectral emitter for visualizing the surgical site. [Figure 12] This graph illustrates an example hyperspectral identification signature for distinguishing the ureter from an obstruction. [Figure 13] This graph illustrates an example of a hyperspectral identification signature used to distinguish arteries from occluders. [Figure 14] This graph illustrates an example of a hyperspectral identification signature used to distinguish nerves from obstructions. [Figure 15]This is a schematic diagram of one embodiment of a near-infrared (NIR) time-of-flight measurement system used during surgery. [Figure 16] Figure 15 shows the time-of-flight timing diagram for the system. [Figure 17] This is a schematic diagram of another embodiment of a near-infrared (NIR) time-of-flight measurement system used during surgery. [Figure 18] This is a schematic diagram of one embodiment of a computer-implemented bidirectional surgical system. [Figure 19] This is a schematic diagram of one embodiment of a surgical system used for performing surgical procedures in an operating room. [Figure 20] This is a schematic diagram of one embodiment of a surgical system including smart surgical instruments and a surgical hub. [Figure 21] Figure 20 is a flowchart showing how to control smart surgical instruments. [Figure 21A] This is a schematic diagram of the colon illustrating major colectomy procedures. [Figure 21B] This is a partial oblique cross-sectional view of one embodiment of a duodenal mucosal surface reconstruction procedure. [Figure 22] This is a schematic diagram of an exemplary surgical system that can provide coordinated control of surgical instruments. [Figure 23] This is an illustrative diagram of an exemplary embodiment of a system for providing a common field of view by extending multiple viewpoints into a common field of view through the use / tracking of sensors placed on multiple endoscopes positioned in the surgical field. [Figure 24] This graph shows the speed versus distance of the surgical instrument shown in Figure 23 as it operates within the patient's intestines. [Figure 25] This is an illustrative diagram of a patient's intestine, identifying the various distances traveled and exemplary rendezvous points at possible surgical sites for the surgical instruments shown in Figure 23. [Figure 26] This is an illustrative diagram of another exemplary embodiment for providing a common field of view by extending multiple viewpoints into a common field of view and by overlaying various exemplary navigation and orientation indicators into the common field of view. [Figure 27] This is an illustrative diagram of an exemplary surgical site in the intestines of a patient, shown in an incorrect orientation. [Figure 28] Figure 23 shows an illustrative diagram of an exemplary surgical site as rotated by the surgical instruments in Figure 27. [Figure 29] This is an illustrative diagram of an example surgical site in Figure 27 after the orientation has been corrected. [Figure 30] Figure 23 is an illustrative diagram of surgical instruments used with a laparoscopic approach. [Modes for carrying out the invention]

[0029] Certain exemplary embodiments are described below so that the structure, function, manufacturing and use principles of the devices, systems, and methods disclosed herein may be understood comprehensively. One or more examples of these embodiments are illustrated in the accompanying drawings. Those skilled in the art will understand that the devices, systems, and methods described in detail herein and shown in the accompanying drawings are non-limiting exemplary embodiments, and that the scope of the invention is defined solely by the claims. Features illustrated or described in relation to one exemplary embodiment may be combined with features of other embodiments. Such modifications and variations shall be within the scope of the invention.

[0030] Furthermore, in this disclosure, components with similar names in embodiments generally have similar characteristics, and therefore, in a particular embodiment, each characteristic of each component with a similar name is not necessarily described in full detail. In addition, to the extent that linear or circular dimensions are used in the description of the disclosed systems, devices, and methods, such dimensions are not intended to limit the types of shapes that can be used in combination with such systems, devices, and methods. Those skilled in the art will recognize that dimensions equivalent to such linear and circular dimensions can be easily determined for any geometric shape. Those skilled in the art will understand that even if the dimensions are not exact values, they can be considered to be close to them due to various factors such as manufacturing tolerances and the sensitivity of measuring instruments. The size and shape of systems and devices, and their components, may depend at least on the size and shape of the components in which the systems and devices are used.

[0031] surgical visualization Generally, surgical visualization systems are configured to leverage “digital surgery” to acquire additional information about a patient’s anatomical structure and / or surgical procedure. Surgical visualization systems are further configured to transmit data to one or more physicians in a useful format. Various aspects of this disclosure use visualization to provide improved visualization of a patient’s anatomical structure and / or surgical procedure, and / or to provide improved control of surgical tools (also referred herein as “surgical devices” or “surgical instruments”).

[0032] "Digital surgery" may include robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, and connectivity both inside and outside the operating room (OR). The various surgical visualization systems described herein can be used in conjunction with robotic surgical systems, but are not limited to use with robotic surgical systems. In certain cases, surgical visualization may occur without a robot, and / or with limited and / or optional robotic assistance. Similarly, digital surgery may occur without a robot, and / or with limited and / or optional robotic assistance.

[0033] In certain cases, surgical systems incorporating surgical visualization systems can enable smart incisions to identify critical structures and avoid them. Critical structures include anatomical structures such as arteries (e.g., ureters, superior mesenteric arteries), veins (e.g., portal veins), nerves (e.g., phrenic nerves), and / or tumors, among other anatomical structures. In other cases, critical structures may be heterogeneous structures within an anatomical region and other heterogeneous structures, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, and plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Smart incision techniques can, for example, provide improved intraoperative guidance for incisions and / or enable smarter decision-making using techniques for detecting and avoiding critical anatomical structures.

[0034] Surgical systems incorporating surgical visualization systems can enable smart anastomosis techniques that provide more consistent anastomoses at optimal locations through improved workflows. Cancer localization diagnostic techniques can be improved using surgical visualization platforms. For example, cancer localization diagnostic techniques can identify and track the location, orientation, and margins of a cancer. In certain cases, cancer localization diagnostic techniques can compensate for the movement of surgical instruments, the patient, and / or the patient's anatomical structures during surgical procedures to provide the physician with guidance to return to the target point.

[0035] Surgical visualization systems can provide improved tissue characterization, as well as lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health status without requiring physical touch, particularly during incision and / or placement of stapling devices within the tissue. Certain tissue characterization techniques can be used without the use of ionizing radiation and / or contrast agents. With regard to lymph node diagnosis and mapping, surgical visualization platforms can, for example, locate, map, and ideally diagnose lymphatic systems and / or lymph nodes involved in cancer diagnosis and staging before surgery.

[0036] During surgical procedures, the information available to the physician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures, such as those embedded or buried within organs, may be at least partially hidden, or even invisible, from the view. Additionally, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or difficult to perceive with the "naked eye." Furthermore, certain structures may move preoperatively (e.g., before the surgical procedure but after the preoperative scan) and / or during the procedure. In such cases, the physician may not be able to accurately determine the location of critical structures during the procedure.

[0037] When the location of critical structures is uncertain, and / or the proximity between critical structures and surgical tools is unknown, the physician's decision-making process may be hindered. For example, a physician may avoid a certain area to avoid inadvertently cutting a critical structure. However, the avoided area may be unnecessarily large and / or at least partially mislocated. Due to uncertainty and / or excessive caution, a physician may fail to reach a specific desired area. For example, even if a critical structure is not in that particular area, and / or the physician's actions in that area would have no adverse effects, over-caution may cause the physician to try to avoid critical structures, leaving behind a portion of the tumor and / or other undesirable tissue. In certain cases, surgical outcomes may improve with increased knowledge and / or certainty, which allows surgeons to be more precise and, in certain cases, more restrained / more aggressive with respect to specific anatomical areas.

[0038] Surgical visualization systems can enable the identification and avoidance of critical structures during surgery. Therefore, surgical visualization systems can enhance intraoperative decision-making and improve surgical outcomes. They can provide advanced visualization capabilities beyond what a physician sees with the naked eye, and / or what imaging systems can perceive and / or communicate to the physician. Surgical visualization systems can improve outcomes in various cases by reinforcing and enhancing what physicians can know before tissue treatment (e.g., incision). As a result, physicians can maintain momentum throughout the surgical procedure, knowing, for example, that the surgical visualization system is tracking critical structures that may be approached during incision. Surgical visualization systems can provide physicians with sufficient time to pause and / or slow down the surgical procedure and assess the proximity to critical structures to prevent accidental damage to them. A surgical visualization system provides physicians with an ideal, optimized, and / or customizable amount of information, enabling them to move confidently and / or quickly throughout the tissue while avoiding accidental damage to healthy tissue and / or critical structures, thereby minimizing the risk of injury resulting from surgical procedures.

[0039] The surgical visualization system is described in detail below. Generally, the surgical visualization system may include a first optical emitter configured to emit multiple spectral waves, a second optical emitter configured to emit a light pattern, and a receiver or sensor configured to detect visible light, molecular responses to spectral waves (spectroscopic imaging), and / or the light pattern. The surgical visualization system may also include an imaging system and a control circuit that signals the receiver and the imaging system. Based on the output from the receiver, the control circuit can determine distances to the surgical site, such as a geometric surface map of the visible surface at the surgical site, e.g., a three-dimensional surface topography, and distances to at least partially hidden structures. The imaging system can communicate the geometric surface map and distances to the physician. In such an example, the augmented view of the surgical site provided to the physician may provide a display of hidden structures in the context related to the surgical site. For example, the imaging system may virtually enhance hidden structures on the geometric surface map of the hidden and / or obstructed tissue, as well as lines drawn on the ground to indicate utility pipes below the surface. Additionally or alternatively, the imaging system can communicate the proximity of a surgical tool to visible obstructing tissue and / or to at least partially hidden structures, and / or the depth of structures hidden below the visible surface of the obstructing tissue. For example, a visualization system can determine the distance to an extended line on the surface of visible tissue and communicate that distance to the imaging system.

[0040] Throughout this disclosure, unless otherwise specified, all references to “light” may include EMR or photons in the visible and / or invisible portions of the electromagnetic radiation (EMR) wavelength spectrum. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by the human eye) and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm. The invisible spectrum (e.g., the non-emission spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum. The invisible spectrum is not detectable by the human eye. Wavelengths longer than about 750 nm are longer than the red visible spectrum and are invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths shorter than about 380 nm are shorter than the violet spectrum and are invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0041] Figure 1 illustrates one embodiment of the surgical visualization system 100. The surgical visualization system 100 is configured to create a visual representation of critical structures 101 within an anatomical region. The critical structures 101 may include one or more critical structures. As discussed herein, the critical structures 101 may be any of the following: anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, blood vessels, tumors, or other anatomical structures; or heterogeneous structures such as surgical devices, surgical fasteners, surgical clips, surgical clasps, bougies, surgical bands, surgical plates, or other heterogeneous structures. As discussed herein, the critical structures 101 may be specific to each patient and / or procedure. Embodiments of critical structures and embodiments of identifying critical structures using the visualization system are further described in U.S. Patent No. 10,792,034, entitled “Visualization of Surgical Devices,” issued on October 6, 2020, which is incorporated herein by reference in its entirety.

[0042] In some examples, the critical structure 101 may be embedded in tissue 103. In other words, tissue 103 can be any of a variety of tissues, such as fat, connective tissue, adhesions, and / or organs. In other words, the critical structure 101 may be located beneath the surface 105 of tissue 103. In such examples, tissue 103 obscures the critical structure 101 from the physician's "gross eye" view. Tissue 103 also obscures the critical structure 101 from the field of view of the imaging device 120 of the surgical visualization system 100. Instead of being completely obscured, the critical structure 101 may be partially obscured from the field of view of the physician and / or the imaging device 120.

[0043] The surgical visualization system 100 can be used in clinical analysis and / or medical intervention. In certain cases, the surgical visualization system 100 can be used intraoperatively to provide a physician with real-time information, such as proximity data, dimensions, and / or distance, during the surgical procedure. Those skilled in the art will recognize that the information is not exactly real-time, but may still be considered real-time due to a variety of reasons, such as time delays caused by data transmission, time delays caused by data processing, and / or the sensitivity of measuring instruments. The surgical visualization system 100 is configured for intraoperative identification of critical structures and / or to facilitate avoidance of critical structures 101 by surgical devices. For example, by identifying critical structures 101, a physician can avoid operating surgical devices around critical structures 101 and / or areas of a predetermined proximity to critical structures 101 during the surgical procedure. In another embodiment, by identifying critical structure 101, the physician can avoid incisions on and / or near critical structure 101, thereby helping to prevent damage to critical structure 101 and / or to surgical devices being damaged by critical structure 101.

[0044] The surgical visualization system 100 is configured to incorporate tissue identification and geometric surface mapping in combination with the surgical visualization system's distance sensor system 104. When combined, these features of the surgical visualization system 100 can determine the location of critical structures 101 within an anatomical region, and / or the proximity of surgical devices 102 to the surface 105 of visible tissue 103 and / or critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system, for example, an imaging device 120 configured to provide a real-time view of the surgical site. The imaging device 120 may include, for example, a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of the image based on molecular responses to various wavelengths. The view from the imaging device 120 may be provided to the physician in real time on a display (e.g., a monitor, computer tablet screen, etc.). The displayed view can be enhanced with additional information based on tissue identification, landscape mapping, and the distance sensor system 104. In such an example, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide the physician with advanced data synthesis and integrated information during surgery.

[0045] The imaging device 120 can be configured to detect, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). Embodiments of the imaging device 120 include scopes, such as endoscopes, arthroscopes, angioscopes, bronchoscopes, common cholangioscopy, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopy (gastroscopy), laryngoscopes, nasopharyngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, or exoscopes. Scopes can be particularly useful in minimally invasive surgical procedures. For open surgery applications, the imaging device 120 may not include a scope.

[0046] The tissue identification subsystem can be achieved using a spectral imaging system. The spectral imaging system may rely on imaging techniques such as hyperspectral imaging, multispectral imaging, or selective spectral imaging. Embodiments of hyperspectral imaging of tissues are further described in U.S. Patent No. 9,274,047, published March 1, 2016, entitled "System and Method for Gross Anatomic Pathology Using Hyperspectral Imaging," which is incorporated herein by reference in its entirety.

[0047] Surface mapping subsystems can be achieved using optical pattern systems. Various surface mapping techniques that use optical patterns (or structured light) for surface mapping can be utilized in the surgical visualization systems described herein. Structured light is the process of projecting a known pattern (often a grid or horizontal bars) onto a surface. In certain cases, invisible (i.e., undetectable) structured light can be utilized, and structured light can be used without interfering with other computer vision tasks where the projected pattern may interfere. For example, interference can be prevented by using infrared light that repeats two exactly opposite patterns or visible light at a very fast frame rate. Embodiments of surgical systems including surface mapping, as well as light sources and projectors for projecting light patterns, are further described in U.S. Patent Application Publication No. 2017 / 0055819, “Set Comprising A Surgical Instrument,” published on 2 March 2017; U.S. Patent Application Publication No. 2017 / 0251900, “Depiction System,” published on 7 September 2017; and U.S. Patent Application No. 16 / 729,751, “Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto,” filed on 30 December 2019, the entirety of which is incorporated herein by reference.

[0048] The distance determination system can be integrated into a surface mapping system. For example, structured light can be used to generate a three-dimensional (3D) virtual model of the visible surface 105, and various distances to the visible surface 105 can be determined. Additionally or alternatively, the distance determination system can rely on time-of-flight measurement to determine one or more distances to specific tissues (or other structures) at the surgical site.

[0049] The surgical visualization system 100 also includes a surgical device 102. The surgical device 102 can be any suitable surgical device. Embodiments of the surgical device 102 include surgical dissection instruments, surgical staplers, surgical grippers, clip applicators, smoke removers, surgical energy devices (e.g., unipolar probes, bipolar probes, ablation probes, ultrasound devices, ultrasound end effectors, etc.). In some embodiments, the surgical device 102 includes an end effector having opposing jaws that extend from the distal end of the shaft of the surgical device 102 and are configured to engage tissue between them.

[0050] The surgical visualization system 100 can be configured to identify critical structures 101 and the proximity of surgical devices 102 to the critical structures 101. The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral camera, a multispectral camera, or a selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral image sensor, detector, and / or three-dimensional camera lens). For example, the imaging device 120 may include right and left lenses used together to generate a three-dimensional (3D) image of the surgical site by simultaneously recording two two-dimensional images, to render a three-dimensional image of the surgical site, and / or to determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicating the topography of visible tissue, as well as the identification and location of hidden important structures, as further described herein. For example, as shown in Figure 1, the field of view of the imaging device 120 can be superimposed on a light pattern (structured light) on the surface 105 of the tissue 103.

[0051] As in this illustrated embodiment, the surgical visualization system 100 can be incorporated into a robotic system 110. The robotic surgical system 110 can have various configurations, as discussed herein. In the illustrated embodiment, the robotic surgical system 110 includes a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 each include a rigid structural member 116 and a joint 118, which can each include servo motor control. The first robotic arm 112 is configured to operate a surgical device 102, and the second robotic arm 114 is configured to operate an imaging device 120. The robotic control unit of the robotic surgical system 110 is configured to generate control motions to the first robotic arm 112 and the second robotic arm 114, which can act on the surgical device 102 and the imaging device 120, respectively.

[0052] In some embodiments, one or more of the robot arms 112, 114 can be separate from the main robot system 110 used in surgical procedures. For example, at least one of the robot arms 112, 114 can be positioned and aligned to a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robot arms 112, 114 can control and / or align the position of the robot arms 112, 114 to a specific coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned to a specific coordinate system.

[0053] Examples of robotic surgical systems include the Ottava® robot-assisted surgical system (Johnson & Johnson of New Brunswick, NJ), the da Vinci® surgical system (Intuitive Surgical, Inc. of Sunnyvale, CA), the Hugo® robot-assisted surgical system (Medtronic PLC of Minneapolis, MN), the Versius® surgical robot system (CMR Surgical Ltd of Cambridge, UK), and the Monarch® platform (Auris Health, Inc. of Redwood City, CA).Various robotic surgical procedures and embodiments using robotic surgical procedures are described in U.S. Patent Application Publication No. 2018 / 0177556, “Flexible Instrument Insertion Using An Adaptive Force Threshold,” filed on December 28, 2016; U.S. Patent Application Publication No. 2020 / 0000530, “Systems And Techniques For Providing Multiple Perspectives During Medical Procedures,” filed on April 16, 2019; U.S. Patent Application Publication No. 2020 / 0170720, “Image-Based Branch Detection And Mapping For Navigation,” filed on February 7, 2020; U.S. Patent Application Publication No. 2020 / 0188043, “Surgical Robotics System,” filed on December 9, 2019; and “Systems And Methods For Concomitant Medical Further details are provided in U.S. Patent Application Publication No. 2020 / 0085516, entitled “Procedures”, U.S. Patent No. 8,831,782, entitled “Patient-Side Surgeon Interface For A Teleoperated Surgical Instrument,” filed on July 15, 2013, and International Publication No. 2014151621, entitled “Hyperdexterous Surgical System,” filed on March 13, 2014.

[0054] The surgical visualization system 100 also includes an emitter 106. The emitter 106 is configured to emit light patterns such as stripes, grid lines, and / or dots to enable determination of the topography or landscape of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from the emitter 106 located on the surgical device 102 and / or one of the robotic arms 112, 114 and / or the imaging device 120. In one embodiment, the projected light array 130 is employed by the surgical visualization system 100 to determine the shape defined during surgery by the surface 105 of the tissue 103 and / or the motion of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0055] As in this illustrated embodiment, the imaging device 120 may include an optical waveform emitter 123, either mounted on the imaging device 120 or otherwise attached. The optical waveform emitter 123 is configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach critical structures 101. The imaging device 120 and the optical waveform emitter 123 may be positionable by a robotic arm 114. The optical waveform emitter 123 may be mounted on the imaging device 122 or otherwise attached, and in other embodiments, it may be positioned on a surgical device separate from the imaging device 120. A corresponding waveform sensor 122 on the imaging device 120 (e.g., an image sensor, spectrometer, or vibration sensor) is configured to detect the effects of electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted from the optical waveform emitter 123 can be configured to enable the identification of anatomical structures and / or types of body structures, such as critical structures 101. Identification of critical structures 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one embodiment, the wavelength of the electromagnetic radiation 124 can be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other embodiments, the waveform sensor 122 and the optical waveform emitter 123 may include, for example, a photoacoustic imaging system.

[0056] The distance sensor system 104 of the surgical visualization system 100 is configured to determine one or more distances at a surgical site. The distance sensor system 104 may be a time-of-flight distance sensor system including an emitter such as the emitter 106 in this illustrated embodiment and a receiver 108. In other examples, the time-of-flight emitter may be separate from the structured light emitter. The emitter 106 may include a very small laser source, and the receiver 108 may include a matching sensor. The distance sensor system 104 is configured to detect "time of flight," or the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, the distance sensor system 104 is able to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104.

[0057] In this illustrated embodiment, the receiver 108 of the distance sensor system 104 is positioned on the surgical device 102, but in other embodiments, the receiver 108 may be mounted on a separate surgical device instead of the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar extending through the surgical device 102 to reach the surgical site. In yet another embodiment, the receiver 108 for the distance sensor system 104 may be mounted on a robotic control arm of a robotic system 110 separate from the first robotic arm 112 to which the surgical device 102 is coupled (e.g., on a second robotic arm 114), on a movable arm operated by another robot, or on an operating room (OR) table or fixture. In some embodiments, the imaging device 120 includes a receiver 108 that allows the distance from the emitter 106 on the surgical device 102 to the surface 105 of the tissue 103 to be determined using a line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, based on the known positions of the emitter 106 (on the surgical device 102) and receiver 108 (on the imaging device 120) of the distance sensor system 104, distance de It is possible to perform triangulation. The three-dimensional position of receiver 108 can be made known with respect to the robot coordinate plane during the operation, and / or aligned.

[0058] As in this illustrated embodiment, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the distance sensor system 104 can be controlled by the second robotic arm 114. In another embodiment, the surgical visualization system 100 may be available separately from the robotic system. In such an example, the distance sensor system 104 may be independent of the robotic system.

[0059] In Figure 1, d e d is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and t d is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue 103. The distance sensor system 104 measures the emitter-tissue distance d e It is configured to determine the device-organizational distance d. t This can be obtained from a known position on the surgical device 102, for example, on its shaft proximal to the distal end of the surgical device 102, relative to the distal end of the surgical device 102. In other words, when the distance between the emitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d can be obtained. t is the emitter-tissue distance d e This can be determined from the following. In some embodiments, the shaft of the surgical device 102 may include one or more articulated joints that can articulate with respect to the emitter 106 and the jaws at the distal end of the surgical device 102. The articulated configuration may include, for example, a multi-articulated vertebral-like structure. In some embodiments, a three-dimensional camera can be used to triangulate one or more distances to the surface 105.

[0060] In Figure 1, d wis the camera-critical structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the critical structure 101, d A is the depth of the critical structure 101 below the surface 105 of the tissue 103 (e.g., the distance between a portion of the surface 105 closest to the surgical device 102 and the critical structure 101). The time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 is the camera-critical structure distance d w configured to determine.

[0061] As shown in Figure 2, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 A is the distance d w , as well as the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (therefore, the known distance d between them x ) determined by triangulation, the distance d e and d A the distance d which is the sum of y can be determined. Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or waveform range) can be used to determine the camera-critical structure distance d w , and a second waveform (or waveform range) can be used to determine the distance to the surface 105 of the tissue 103. In such an example, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0062] Additionally or alternatively, the distance d A can be determined from ultrasound, registered magnetic resonance imaging (MRI), or computerized tomography (CT) scans. In still other examples, the distance d ASince the detection signal received by the imaging device 120 may vary based on the type of material, for example, the type of tissue 103, it can be determined by spectral imaging. For example, fat can reduce the detection signal by a first method or by a first amount, and collagen can reduce the detection signal by a different second method or by a second amount.

[0063] In another embodiment of the surgical visualization system 160 illustrated in Figure 3, the surgical device 162, rather than the imaging device 120, includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform emitter 123 is located at a distance d from a common device such as the surgical device 162, as further described herein. t , and d w It is configured to emit a waveform for determining the distance d from the surface 105 of tissue 103 to the surface of critical structure 101. A This can be determined as follows: d A =d w -d t

[0064] The surgical visualization system 100 includes a control system configured to control various aspects of the surgical visualization system 100. Figure 4 illustrates one embodiment of a control system 133 that can be used as a control system for the surgical visualization system 100 (or other surgical visualization systems described herein). The control system 133 includes a control circuit 132 configured to signal-communicate with a memory 134. The memory 134 is configured to store instructions that can be executed by the control circuit 132, such as instructions for determining and / or recognizing critical structures (e.g., critical structure 101 in Figure 1), instructions for determining and / or calculating one or more distances and / or three-dimensional digital representations, and instructions for communicating information to a physician. Thus, the instructions stored in the memory 134 constitute a computer program product that, when executed by the processor, includes instructions that cause the processor to execute as described above. Such instructions may also be stored on any computer-readable medium (such as an optical disc, SD card, USB drive, or memory of a separate device), copied from there to the memory 134, or executed directly. The copy or direct execution process involves creating a data carrier signal that carries the computer program product. As in this illustrated embodiment, memory 134 can store surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141, but memory 134 can store any combination of logic 136, 138, 140, and 141, and / or various logics can be combined together. Control system 133 also includes imaging system 142 (e.g., imaging system including imaging device 120 in Figure 1) including camera 144, display 146 (e.g., monitor, computer tablet screen), and control unit 148 for camera 144 and display 146. Camera 144 includes image sensor 135 (e.g., waveform sensor 122) (e.g., visible light, spectral imager, three-dimensional lens, etc.) configured to receive signals from various light sources emitting light in various visible and invisible spectra.The display 146 is configured to show the physician real, virtual, and / or virtually augmented images and / or information.

[0065] In exemplary embodiments, the image sensor 135 is a solid-state electronic device containing up to several million individual photodetector segments called pixels. The technology of the image sensor 135 is classified into one of two categories: charge-coupled device (CCD) imagers and complementary metal oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being a more recent development in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (marketed as "sCOMS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. The CCD and CMOS image sensors 135 are sensitive to wavelengths in the range of approximately 350 nm to 1050 nm, such as in the range of approximately 400 nm to 1000 nm. Those skilled in the art will recognize that while a value is not precisely the same as the actual value, it may still be considered approximately that value due to various reasons such as the sensitivity of the measuring instrument and manufacturing tolerances. CMOS sensors are generally more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Therefore, there are two types of color CCD cameras: one-chip and three-chip. One-chip color CCD cameras offer a commonly adopted, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to separate incident light into a series of colors and employing interpolation algorithms to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing prisms, directing each section of the incident spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value rather than using an algorithm to determine the color. Three-chip cameras offer very high resolution.

[0066] The control system 133 also includes emitters (e.g., emitter 106) each containing a spectral light source 150 and a structured light source 152, each operably coupled to the control circuit 133. A single light source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, a single light source can be pulsed to provide light in the visible spectrum (e.g., infrared spectral light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. The tissue identification logic 140 is configured to identify important structures (e.g., important structure 101 in Figure 1) via data from the spectral light source 150 received by the image sensor 135 of the camera 144. The surface mapping logic 136 is configured to determine the surface contour of a visible tissue (e.g., tissue 103) based on reflected structured light. By measuring time of flight, the distance determination logic 141 is configured to determine the distance to one or more visible tissues and / or critical structures. The outputs from each of the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 are provided to the imaging logic 138, which is configured to combine, integrate, and / or superimpose them so that they are communicated to the physician via the display 146 of the imaging system 142.

[0067] The control circuit 132 can have various configurations. Figure 5 illustrates one embodiment of a control circuit 170 that can be used as a control circuit 132 configured to control aspects of the surgical visualization system 100. The control circuit 170 is configured to implement various processes described herein. The control circuit 170 includes a microcontroller, which includes a processor 172 (e.g., a microprocessor or microcontroller) operably coupled to a memory 174. The memory circuit 174 is configured to store machine-executable instructions, which, when executed by the processor 172, cause the processor 172 to execute machine instructions for implementing various processes described herein. The processor 172 can be any one of several single-core or multi-core processors known in the art. The memory circuit 174 can include volatile and non-volatile storage media. The processor 172 includes an instruction processing unit 176 and an arithmetic unit 178. The instruction processing unit 176 is configured to receive instructions from the memory circuit 174.

[0068] The surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141 can have various configurations. Figure 6 illustrates one embodiment of a combinational logic circuit 180 configured to control aspects of the surgical visualization system 100 using logic such as one or more of the surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141. The combinational logic circuit 180 includes a finite state machine including a combinational logic 182 configured to receive data associated with surgical devices (e.g., surgical device 102 and / or imaging device 120) at input 184, process the data by the combinational logic 182, and provide the output 184 to a control circuit (e.g., control circuit 132).

[0069] Figure 7 illustrates one embodiment of a sequential logic circuit 190 configured to control aspects of a surgical visualization system 100 using logic such as one or more of the following: surface mapping logic 136, imaging logic 138, tissue identification logic 140, and distance determination logic 141. The sequential logic circuit 190 includes a finite state machine comprising combinational logic 192, memory 194, and a clock 196. The memory 194 is configured to store the current state of the finite state machine. The sequential logic circuit 190 can be synchronous or asynchronous. The combinational logic 192 is configured to receive data associated with surgical devices (e.g., surgical device 102 and / or imaging device 120) at input 426, process the data by the combinational logic 192, and provide an output 499 to a control circuit (e.g., control circuit 132). In some embodiments, the sequential logic circuit 190 may include a combination of a processor (e.g., processor 172 in Figure 5) and a finite state machine implementing the various processes described herein. In some embodiments, a finite state machine may include a combination of combinational logic circuits (e.g., combinational logic circuit 192 in Figure 7) and sequential logic circuits 190.

[0070] Figure 8 illustrates another embodiment of the surgical visualization system 200. The surgical visualization system 200 is generally configured and used similarly to the surgical visualization system 100 of Figure 1, and includes, for example, a surgical device 202 and an imaging device 220. The imaging device 220 includes, for example, a spectral light emitter 223 configured to emit spectral light of multiple wavelengths to acquire spectral images of hidden structures. The imaging device 220 may also include a three-dimensional camera and associated electronic processing circuits. The surgical visualization system 200 is shown being used in surgery to identify certain critical structures, such as the ureter 201a which is invisible on the surface 205 of organ 203, and blood vessels 201b within organ 203 (in this example, the uterus), and to facilitate their avoidance.

[0071] The surgical visualization system 200 uses structured light to measure the emitter-tissue distance d from the emitter 206 on the surgical device 202 to the surface 205 of the uterus 203. e The surgical visualization system 200 is configured to determine the emitter-tissue distance d e Based on this, the device-tissue distance d from the surgical device 202 to the surface 205 of the uterus 203 t It is configured to extrapolate. The surgical visualization system 200 also calculates the tissue-ureteral distance d from the ureter 201a to the surface 205. A , and the camera-ureteral distance d from the imaging device 220 to the ureter 201a w It is configured to determine the distance d. As described herein, for example, with respect to the surgical visualization system 100 of Figure 1, the surgical visualization system 200 uses, for example, spectral imaging and time-of-flight sensors to determine the distance d w It is configured to determine the tissue-ureteral distance d based on other distances and / or surface mapping logic described herein. In various embodiments, the surgical visualization system 200 determines the tissue-ureteral distance d based on other distances and / or surface mapping logic described herein. A (That is, depth) can be determined (for example, by triangulation).

[0072] As described above, the surgical visualization system includes a control system configured to control various aspects of the surgical visualization system. The control system can have various configurations. Figure 9 illustrates one embodiment of a control system 600 for a surgical visualization system, such as the surgical visualization system 100 of Figure 1, the surgical visualization system 200 of Figure 8, or other surgical visualization systems described herein. The control system 600 is a transformation system that integrates spectral signature tissue identification and structured phototissue positioning to detect tissue variability, in particular when important structures are obscured by tissues, e.g., fat, connective tissue, hematopoietic tissue, and / or organs, and / or blood, and / or distinguish tumors and / or non-healthy tissues from healthy tissues within organs.

[0073] The control system 600 is configured to implement a hyperspectral imaging visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field. The control system 600 includes a conversion logic circuit 648 configured to convert tissue data into information usable by surgeons and / or other healthcare professionals. For example, key structures within an anatomical structure can be identified using wavelength-based variable reflectivity to obscuring materials. Furthermore, the control system 600 is configured to combine identified spectral signatures with structural optical data within an image. For example, the control system 600 can be employed to create a three-dimensional dataset for surgical applications in a system using augmented image overlays. The technology can be employed both during and before surgery, incorporating additional visual information. In various embodiments, the control system 600 is configured to provide alerts to the physician when one or more key structures are in proximity. Various algorithms can be employed to guide robotically automated and semi-automated approaches based on surgical procedures and proximity to key structures.

[0074] The projected light array is employed by the control system 600 to determine the shape and movement of the tissue during surgery. Alternatively, flash lidar may be used for tissue surface mapping.

[0075] As described above, the control system 600 is configured to detect critical structures, which may include one or more critical structures, provide an image overlay of the critical structures, and measure the distance to the surface of the visible tissue and the distance to embedded / buried critical structures. The control system 600 can measure the distance to the surface of the visible tissue, or detect critical structures and provide an image overlay of the critical structures.

[0076] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 may be a field programmable gate array (FPGA) or another preferred circuit configuration, such as the configuration described with respect to Figures 6, 7, and 8. The spectral control circuit 602 includes a processor 604 configured to receive a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, and for example, C / C++ code may be used. The video input processor 606 is configured to receive video in control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608, which includes, for example, a hyperspectral video out of interface control (metadata) data. The video output processor 608 is configured to provide the video output signal to an image overlay controller 610.

[0077] The video input processor 606 is operably coupled to the patient-side camera 612 via a patient isolation circuit 614. The camera 612 includes a solid-state image sensor 634. The patient isolation circuit 614 may include a number of transformers to isolate the patient from other circuits in the system. The camera 612 is configured to receive intraoperative images via an optical element 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor or another image sensor technology such as those discussed herein in relation to Figure 4. The camera 612 is configured to output 613 images with a 14-bit / pixel signal. Those skilled in the art will recognize that higher or lower resolutions may be employed. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which in this illustrated embodiment employs a camera hardware register 618 and a Nios2 coprocessor 620 configured to process the camera output signal 613. The video input processor 606 and the laser pulse control circuit 622 are provided with a color RGB fused output signal.

[0078] The laser pulse control circuit 622 is configured to control the laser light engine 624. The laser light engine 624 is configured to output light at multiple wavelengths (λ1, λ2, λ3...λn), including near-infrared (NIR). The laser light engine 624 can operate in multiple modes. For example, the laser light engine 624 can operate in two modes. In a first mode, for example, the standard operating mode, the laser light engine 624 is configured to output an illumination signal. In a second mode, for example, the specific mode, the laser light engine 624 is configured to output RGBG light and NIR light. In various embodiments, the laser light engine 624 can operate in polarization mode.

[0079] The optical output 626 from the laser light engine 624 is configured to illuminate a targeted anatomical structure within the surgical site 627 during surgery. The laser pulse control circuit 622 is also configured to control a laser pulse controller 628 for a laser pattern projector 630 configured to project a laser light pattern 631, such as a grid or pattern of lines and / or dots, at a predetermined wavelength (λ2) onto surgical tissue or organs in the surgical site 627. The camera 612 is configured to receive the patterned and reflected light output through the camera optical element 632. The image sensor 634 is configured to convert the received light into a digital signal.

[0080] The color RGB fusion circuit 616 is also configured to output signals to the image overlay controller 610 and to a video input module 636 for reading the laser beam pattern 631 projected by the laser pattern projector 630 onto the targeted anatomical structure at the surgical site 627. The processing module 638 is configured to process the laser beam pattern 631 and output a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 is also configured to output a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the targeted anatomical structure at the surgical site.

[0081] The first video output signal 640 and the second video output signal 642 include data representing the location of important structures on a three-dimensional surface model provided to the integrated module 643. Combined with the data from the video output processor 608 of the spectral control circuit 602, the integrated module 643 calculates the distance to the buried important structures (e.g., distance d in Figure 1). AThe system is configured to determine (for example, by a triangulation algorithm 644) the distance to the buried critical structure, which can be provided to the image overlay controller 610 via the video output processor 646. The aforementioned transformation logic may include a transformation logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0082] Preoperative data 650 obtained from CT or MRI scans can be used to align or position specific three-dimensionally deformable tissues in various examples. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610, so that this information can be superimposed on the view from the camera 612 and provided to the video monitor 652. Embodiments of preoperative data alignment are further described in U.S. Patent Application Publication No. 2020 / 0015907, “Integration of Imaging Data,” filed September 11, 2018, which is incorporated herein by reference in its entirety.

[0083] The video monitor 652 is configured to output an integrated / enlarged view from the image overlay controller 610. The physician can select and / or switch between different views on one or more displays. In an illustrated embodiment, on a first display 652a, which is a monitor, the physician can switch between (A) a view showing a three-dimensional rendering of visible tissue and (B) an enlarged view in which one or more hidden critical structures are depicted on top of the three-dimensional rendering of visible tissue. In an illustrated embodiment, on a second display 652b, which is a monitor, the physician can switch between, for example, one or more hidden critical structures and / or distance measurements to the surface of visible tissue.

[0084] Various surgical visualization systems described herein can be used to visualize various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be obscured by EMR in the visible portion of the spectrum. The surgical visualization system can utilize a spectral imaging system, as described above, which can be configured to visualize different types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the tissue's absorption coefficients across various EMR wavelengths. The spectral imaging system can be configured to characterize the tissue type of the visualized tissue based on a specific combination of constituent materials.

[0085] Figure 10 shows Graph 300 illustrating how the absorption coefficients of various biomaterials vary across the EMR wavelength spectrum. In Graph 300, the vertical axis 302 represents the absorption coefficient of the biomaterial in cm². -1 The units are expressed, and the horizontal axis 304 represents the EMR wavelength in μm. In this graph 300, the first line 306 represents the absorption coefficient of water at various EMR wavelengths, the second line 308 represents the absorption coefficient of protein at various EMR wavelengths, the third line 310 represents the absorption coefficient of melanin at various EMR wavelengths, the fourth line 312 represents the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, the fifth line 314 represents the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and the sixth line 316 represents the absorption coefficient of collagen at various EMR wavelengths. Since different tissue types have different combinations of constituent materials, the tissue types visualized by the surgical visualization system can be identified and differentiated according to specific combinations of constituent materials detected. Therefore, the spectral imaging system of a surgical visualization system can be configured to emit EMR at several different wavelengths, determine the constituent materials of the tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on a specific detected combination of constituent materials.

[0086] Figure 11 illustrates one embodiment of the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures. In Figure 11, a spectral emitter 320 (e.g., spectral light source 150 in Figure 4) is used by the imaging system to visualize the surgical site 322. EMR emitted by the spectral emitter 320 and reflected from the tissues and / or structures of the surgical site 322 is received by an image sensor (e.g., image sensor 135 in Figure 4) to visualize the tissues and / or structures, which may be either visible (e.g., located on the surface of the surgical site 322) or obscured (e.g., beneath other tissues and / or structures in the surgical site 322). In this embodiment, the imaging system 142 (e.g., the imaging system in Figure 4) visualizes tumors 324, arteries 326, and various abnormalities 328 (e.g., tissues that cannot be identified against known or expected spectral signatures) for each of different tissue / structure types, based on spectral signatures characterized by different absorption properties (e.g., absorption coefficients) of the constituent materials. The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system (e.g., display 146 of the imaging system 142 in Figure 4), on a main display (e.g., the main display 819 in Figure 19), on a non-sterile display (e.g., non-sterile displays 807, 809 in Figure 19), on a surgical hub display (e.g., the display of the surgical hub 806 in Figure 19), on a device / instrument display, and / or on another display.

[0087] The imaging system can be configured to adjust or update the visualization of the displayed surgical site according to the identified tissue and / or structural type. For example, as shown in Figure 11, the imaging system can display a margin 330 associated with a tumor 324 visualized on a display screen associated with or coupled to the imaging system, on the main display, on a non-sterile display, on the surgical hub display, on the device / instrument display, and / or on another display. The margin 330 can indicate the area or amount of tissue to be excised to ensure complete removal of the tumor 324. The control system of the surgical visualization system (e.g., control system 133 in Figure 4) can be configured to control or update the dimensions of the margin 330 based on the tissue and / or structure identified by the imaging system. In the illustrated embodiment, the imaging system identifies several anomalies 328 within the field of view (FOV). The control system can then adjust the displayed margin 330 to a first updated margin 332 having sufficient dimensions to encompass these anomalies 328. Furthermore, the imaging system identifies artery 326 that partially overlaps with the initially displayed margin 330 (as indicated by the highlighted region 334 of artery 326). Therefore, the control system can adjust the displayed margin to a second updated margin 336 that has sufficient dimensions to encompass the relevant portion of artery 326.

[0088] Tissues and / or structures can also be imaged or characterized according to these reflectivity characteristics across the EMR wavelength spectrum, in addition to or instead of the absorptivity characteristics described above with respect to Figures 10 and 11. For example, Figures 12, 13, and 14 illustrate various graphs of the reflectivity of different types of tissues or structures across various EMR wavelengths. Figure 12 is a graph representation of an exemplary ureteral signature against an obscuring object. Figure 13 is a graph representation of an exemplary arterial signature against an obscuring object. Figure 14 is a graph representation of an exemplary nerve signature against an obscuring object. The plots in Figures 12, 13, and 14 show the reflectivity of specific structures (ureters, arteries, and nerves) as a function of wavelength (nm) against the corresponding reflectivity of fat, lung tissue, and blood at the corresponding wavelengths. These graphs are for illustrative purposes only, and it should be understood that other organizations and / or structures may include corresponding detectable reflectance signatures that enable identification and visualization of the organization and / or structure.

[0089] Selective wavelengths for spectral imaging can be identified and utilized based on anticipated important structures and / or occluding elements at the surgical site (e.g., "selective spectral" imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized so that information can be acquired in real time and used during surgery. Wavelengths can be selected by the physician or by the control circuit based on user input, e.g., from the physician. In certain examples, wavelengths can be selected based on machine learning and / or big data accessible to the control circuit, e.g., via the cloud or surgical hub.

[0090] Figure 15 illustrates one embodiment of spectral imaging of tissue used in surgery to measure the distance between a waveform emitter and a critical structure obscured by tissue. Figure 15 shows one embodiment of a time-of-flight sensor system 404 utilizing waveforms 424, 425. The time-of-flight sensor system 404 can be incorporated into a surgical visualization system, for example, as sensor system 104 of the surgical visualization system 100 in Figure 1. The time-of-flight sensor system 404 includes a waveform emitter 406 and a waveform receiver 408 on the same surgical device 102 (e.g., emitter 106 and receiver 108 on the same surgical device 402 in Figure 1). The emitted wave 400 extends from the emitter 406 to the critical structure 401 (e.g., critical structure 101 in Figure 1), and the received wave 425 is reflected back from the critical structure 401 to the receiver 408. In this illustrated embodiment, the surgical device 402 is positioned through a trocar 410 that extends into the patient's cavity 407. In this illustrated embodiment, a trocar 410 is used, but other trocars or other access devices may be used, or no access device may be used at all.

[0091] Waveforms 424 and 425 are configured to penetrate the occluding tissue 403, for example, by having wavelengths within the NIR or SWIR spectrum. A spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal is emitted from the emitter 406, as indicated by a first arrow 407 pointing distally, and can penetrate the tissue 403 in which the critical structure 401 is occluded. The emitted waveform 424 is reflected by the critical structure 401, as indicated by a second arrow 409 pointing proximal. The received waveform 425 may be delayed due to the distance d between the distal end of the surgical device 402 and the critical structure 401. Waveforms 424 and 425 can be selected to target the critical structure 401 in the tissue 403 based on the spectral signature of the critical structure 401, as described herein. The emitter 406 is configured to provide a binary signal representing on and off, as shown in Figure 16, for example, and can be measured by the receiver 408.

[0092] Based on the delay between the emitted wave 424 and the received wave 425, the time-of-flight sensor system 404 is configured to determine the distance d. The time-of-flight timing diagram 430 of the emitter 406 and receiver 408 in Figure 15 is shown in Figure 16. The delay is a function of the distance d, and the distance d is determined by:

[0093]

number

[0094] The time of flight of waveforms 424 and 425 corresponds to distance d in Figure 15. In various examples, an additional emitter / receiver and / or pulse signal from emitter 406 can be configured to emit an opaque signal. The opaque signal can be configured to determine the distance from emitter 406 to the surface 405 of the covering tissue 403. In various examples, the depth of critical structure 401 can be determined by: d A =d w -d t In the formula, d A =This is the depth of important structure 401, d w = This is the distance from emitter 406 to critical structure 401 (d in Figure 15), and d t = This is the distance from the emitter 406 (on the distal end of the surgical device 402) to the surface 405 of the covering tissue 403.

[0095] Figure 17 illustrates another embodiment of a time-of-flight sensor system 504 utilizing waves 524a, 524b, 524c, 525a, 525b, and 525c. The time-of-flight sensor system 504 can be incorporated into a surgical visualization system, for example, as sensor system 104 in the surgical visualization system 100 of Figure 1. The time-of-flight sensor system 504 includes a waveform emitter 506 and a waveform receiver 508 (e.g., emitter 106 and receiver 108 in Figure 1). The waveform emitter 506 is positioned on a first surgical device 502a (e.g., surgical device 102 in Figure 1), and the waveform receiver 508 is positioned on a second surgical device 502b. The surgical devices 502a and 502b are positioned through a first trocar 510a and a second trocar 510b, respectively, extending into the patient's cavity 507. While trocars 510a and 510b are used in this illustrated embodiment, other trocars or other access devices may be used, or no access device may be used at all. The emitted waves 524a, 524b, and 524c extend from the emitter 506 toward the surgical site, and the received waves 525a, 525b, and 525c are reflected from the receiver 508 by various structures and / or surfaces at the surgical site.

[0096] The different emitted waves 524a, 524b, and 524c are configured to target different types of material at the surgical site. For example, wave 524a targets the covering tissue 503, wave 524b targets a first critical structure 501a (e.g., critical structure 101 in Figure 1) which is a blood vessel in this illustrated embodiment, and wave 524c targets a second critical structure 501b (e.g., critical structure 101 in Figure 1) which is a cancerous tumor in this illustrated embodiment. The wavelengths of waves 524a, 524b, and 524c can be wavelengths of visible light, NIR, or SWIR spectra. For example, visible light can be reflected by the surface 505 of the tissue 503, and NIR and / or SWIR waveforms can be transmitted through the surface 505 of the tissue 503. In various embodiments, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from the emitter 506, as described herein. Waves 524b, 524c can be selected to target key structures 501a, 501b within tissue 503 based on the spectral signatures of key structures 501a, 501b, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, which are incorporated herein by reference.

[0097] The emitted waves 524a, 524b, and 524c are reflected from the target material, namely the surface 505, the first critical structure 501a, and the second structure 501b, respectively. The received waveforms 525a, 525b, and 525c are reflected at a distance d 1a d 2a d 3a d 1b d 2b d 2c There is a possibility of delays due to this.

[0098] In a time-of-flight sensor system 504 in which the emitter 506 and receiver 508 can be positioned individually (for example, on separate surgical devices 502a, 502b and / or controlled by a separate robotic arm), various distances d 1a d 2a d 3a d 1b d 2b d 2c This can be calculated from the known positions of the emitter 506 and receiver 508. For example, when surgical devices 502a and 502b are robotically controlled, their positions can be known. With knowledge of the positions of the emitter 506 and receiver 508, as well as the time of the photon stream targeting a specific tissue and the information of that specific response received by receiver 508, the distance d can be calculated. 1a d 2a d 3a d 1b d 2b d 2c This makes it possible to determine the distance. In one embodiment, the distance to the obscured critical structures 501a, 501b can be determined by triangulation using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 504 can determine various distances.

[0099] The view provided to the physician, for example on a display, allows the receiver 508 to be rotated in a plane perpendicular to the axis of a selected target structure 503, 501a, or 501b, such that the center of mass of the target structure in the resulting image remains constant. Such orientations can quickly communicate one or more associated distances and / or viewpoints with respect to the target structure. For example, as shown in Figure 17, the surgical site is displayed from a viewpoint where the critical structure 501a is perpendicular to the view plane (e.g., blood vessels are facing inward and outward from the page). Such an orientation can be the default setting. However, the view may be rotated or otherwise adjusted by the physician. In certain examples, the physician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0100] As in this illustrated embodiment, the receiver 508 can be mounted on a trocar 510b (or other access device) through which the surgical device 502b is positioned. In other embodiments, the receiver 508 can be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 508 can be mounted on a movable arm separate from the robot controlling the surgical device 502a, or on an operating room (OR) table or fixture that can be intraoperatively aligned to the robot coordinate plane. In such examples, the positions of the emitter 506 and the receiver 508 can be aligned to the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 504.

[0101] The combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS), known as TOF-NIRS, which enables the measurement of time-resolved profiles of NIR light with nanosecond resolution, can be found in *Time-Of-Flight Near-Infrared Spectroscopy For Nondestructive Measurement Of Internal Quality In Grapefruit*, *Journal of the American Society for Horticultural Science*, May 2013, vol. 138, no. 3, pp. 225-228, which is incorporated herein by reference in its entirety.

[0102] The visualization system, its forms, and embodiments of use are described in U.S. Patent Application Publication No. 2020 / 0015923, “Surgical Visualization Platform,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015900, “Controlling An Emitter Assembly Pulse Sequence,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015668, “Singular EMR Source Emitter Assembly,” filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015925, “Combination Emitter And Camera Assembly,” filed on September 11, 2018; and “Surgical Visualization With Proximity Tracking U.S. Patent Application Publication No. 2020 / 00015899, entitled "Features", U.S. Patent Application Publication No. 2020 / 00015903, entitled "Surgical Visualization Of Multiple Targets", filed on September 11, 2018, and U.S. Patent Application Publication No. 10,792, entitled "Visualization Of Surgical Devices", filed on September 11, 2018.U.S. Patent Application Publication No. 034, entitled "Operative Communication Of Light" (2020 / 0015897), filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015924, entitled "Robotic Light Projection Tools", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015898, entitled "Surgical Visualization Feedback System", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015906, entitled "Surgical Visualization And Monitoring", filed on September 11, 2018; U.S. Patent Application Publication No. 2020 / 0015907, entitled "Integration Of Imaging Data", filed on September 11, 2018; "Robotically-Assisted Surgical U.S. Patent No. 10,925,598, titled "Surgical Surgical Systems"; U.S. Patent Publication No. 2020 / 0015901, filed September 11, 2018, titled "Safety Logic For Surgical Surgical Systems"; U.S. Patent Publication No. 2020 / 0015914, filed September 11, 2018, titled "Robotic Systems With Separate Photoacoustic Receivers"; U.S. Patent Publication No. 2020 / 0015902, filed September 11, 2018, titled "Force Sensor Through Structured Light Deflection"; U.S. Patent Publication No. 2019 / 0201136, filed December 4, 2018, titled "Method Of Hub Communication"; U.S. Patent Publication No. 2019 / 0201136, filed December 30, 2019, titled "Analyzing Surgical Trends By A Surgical U.S. Patent Application No. 16 / 729,772, titled "System", and U.S. Patent Application No. 16 / 729, filed on December 30, 2019, titled "Dynamic Surgical Visualization Systems",U.S. Patent Application No. 16 / 729,744, filed December 30, 2019, entitled "Visualization Systems Using Structured Light"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue"; U.S. Patent Application No. 16 / 729,729, filed December 30, 2019, entitled "Surgical Systems For Proposing And Corroborating Organ Portion Removals"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ"; U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical Systems For Generating Three U.S. Patent Application No. 16 / 729,751 entitled "Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto", U.S. Patent Application No. 16 / 729,740 entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data", filed on December 30, 2019, and U.S. Patent Application No. 16 / 729,740 entitled "Adaptive Surgical System Control According To Surgical Smoke Cloud Characteristics", filed on December 30, 2019.U.S. Patent Application No. 16 / 729,796, filed December 30, 2019, entitled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics"; U.S. Patent Application No. 16 / 729,803, filed December 30, 2019, entitled "Adaptive Visualization By A Surgical System"; U.S. Patent Application No. 16 / 729,807, filed December 30, 2019, entitled "Method Of Using Imaging Devices In Surgery"; U.S. Patent Application No. 63 / 249,644, filed September 29, 2021, entitled "Surgical Devices, Systems, And Methods Using Multi-Source Imaging"; U.S. Patent Application No. 63 / 249,644, filed September 29, 2021, entitled "Surgical Devices, Systems, Methods Using Fiducial Identification And Further details are provided in U.S. Patent Application No. 63 / 249,652 entitled "Tracking," U.S. Patent Application No. 63 / 249,658 entitled "Surgical Devices, Systems, and Methods For Control Of One Visualization With Another," filed September 29, 2021, U.S. Patent Application No. 63 / 249,877 entitled "Methods and Systems For Controlling Cooperative Surgical Instruments," filed September 29, 2021, and U.S. Patent Application No. 63 / 249,980 entitled "Cooperative Access," filed September 29, 2021, which are incorporated herein by reference in their entirety.

[0103] Surgical Hub The various visualization or imaging systems described herein can be incorporated into a system including a surgical hub. Generally, a surgical hub can span multiple healthcare facilities and be a component of a comprehensive digital healthcare system configured to provide integrated, comprehensive, and improved healthcare to a vast number of patients. The comprehensive digital healthcare system includes a cloud-based healthcare analytics system configured to interconnect multiple surgical hubs located across many different healthcare facilities. A surgical hub is configured to interconnect one or more elements, such as one or more surgical instruments used to perform medical procedures on patients and / or one or more visualization systems used during the performance of medical procedures. The surgical hub provides diverse functionality to improve the outcomes of medical procedures. Data on patients and medical procedures generated by various surgical devices, visualization systems, and surgical hubs can be transmitted to the cloud-based healthcare analytics system. This data can then be aggregated with similar data collected from many other surgical hubs, visualization systems, and surgical instruments located in other healthcare facilities. Various patterns and correlations can be found through the cloud-based analytics system that analyzes the collected data. As a result, improvements in the techniques used to generate data can be made, and these improvements can then be disseminated to various surgical hubs, visualization systems, and surgical devices. The interconnectivity of all the components described above may lead to improvements in medical procedures and practices that might not be found if many of the components were not interconnected in this way.

[0104] Examples of surgical hubs configured to receive, analyze, and output data, and methods for using such surgical hubs, are described in U.S. Patent Application Publication No. 2019 / 0200844, “Method Of Hub Communication, Processing, Storage And Display,” filed December 4, 2018; U.S. Patent Application Publication No. 2019 / 0200981, “Method Of Compressing Tissue Within A Stapling Device And Simultaneously Displaying The Location Of The Tissue Within The Jaws,” filed December 4, 2018; U.S. Patent Application Publication No. 2019 / 0201046, “Method For Controlling Smart Energy Devices,” filed December 4, 2018; and “Adaptive Control Program Updates For Surgical U.S. Patent Application Publication No. 2019 / 0201114 titled "Hubs", U.S. Patent Application Publication No. 2019 / 0201140 titled "Surgical Hub Situational Awareness" filed on March 29, 2018, U.S. Patent Application Publication No. 2019 / 0206004 titled "Interactive Surgical Systems With Condition Handling Of Devices And Data Capabilities" filed on March 29, 2018, U.S. Patent Application Publication No. 2019 / 0206555 titled "Cloud-based Medical Analytics For Customization And Recommendations To A User" filed on March 29, 2018, and "Surgical Network Determination Of Prioritization Of Communication, Interaction,Further details are provided in U.S. Patent Application Publication 2019 / 0207857, entitled “Or Processing Based On System Or Device Needs,” which are incorporated herein by reference in their entirety.

[0105] Figure 18 illustrates one embodiment of a computer-implemented bidirectional surgical system 700, which includes one or more surgical systems 702 and a cloud-based system (e.g., a cloud 704 which may include a remote server 713 coupled to a storage device 705). Each surgical system 702 includes at least one surgical hub 706 that communicates with the cloud 704. In one embodiment, as illustrated in Figure 18, the surgical system 702 includes a visualization system 708, a robotic system 710, and intelligent (or "smart") surgical instruments 712, which are configured to communicate with each other and / or with the hub 706. The intelligent surgical instruments 712 may include an imaging device. The surgical system 702 may include M hubs 706, N visualization systems 708, O robotic systems 710, and P intelligent surgical instruments 712, where M, N, O, and P are one or more integers which may be equal to or not equal to one or more of each other. Various exemplary intelligent surgical instruments and robotic systems are described herein.

[0106] The data received by the surgical hub from the surgical visualization system can be used in any of the following ways. In an exemplary embodiment, the surgical hub can receive data from, for example, a surgical visualization system used in the operating room during the execution of a surgical procedure, or used with the patient in the surgical setting. The surgical hub can use the data received in any one or more ways, as discussed herein.

[0107] The surgical hub, in conjunction with the use of the surgical visualization system, can be configured to analyze received data in real time and, based on the analysis of the received data, to adjust and control one or more of the surgical visualization system and / or one or more intelligent surgical instruments used with the patient. Such adjustments may include, for example, adjusting one or more operational control parameters of the intelligent surgical instrument, having one or more sensors of one or more intelligent surgical instruments take measurements to help understand the patient's current physiological state and / or the current operational state of the intelligent surgical instrument, and other adjustments. Controlling and adjusting the operation of intelligent surgical instruments will be discussed further below. Examples of operational control parameters for intelligent surgical instruments include motor speed, cutting element speed, time, duration, energy application level, and light emission. Examples of surgical hubs, as well as examples of controlling and regulating the operation of intelligent surgical instruments, are described in U.S. Patent Application No. 16 / 729,772, “Analyzing Surgical Trends By A Surgical System,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,747, “Dynamic Surgical Visualization Systems,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,744, “Visualization Systems Using Structured Light,” filed December 30, 2019; U.S. Patent Application No. 16 / 729,778, “System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue,” filed December 30, 2019; and “Surgical Systems For Proposing And Corroborating Organ Portion U.S. Patent Application No. 16 / 729, entitled “Removals”,U.S. Patent Application No. 16 / 729,778, filed December 30, 2019, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ"; U.S. Patent Application No. 16 / 729,751, filed December 30, 2019, entitled "Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto"; U.S. Patent Application No. 16 / 729,740, filed December 30, 2019, entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data"; and "Adaptive Surgical System Control According To Surgical Smoke Cloud". U.S. Patent Application No. 16 / 729,737, titled "Characteristics", U.S. Patent Application No. 16 / 729,796, filed December 30, 2019, titled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics", U.S. Patent Application No. 16 / 729,803, filed December 30, 2019, titled "Adaptive Visualization By A Surgical System", and U.S. Patent Application No. 16 / 729, filed December 30, 2019, titled "Method Of Using Imaging Devices In Surgery".Further details are provided in Patent No. 807. These include U.S. Patent Application No. 17 / 068,857, “Adaptive Responses From Smart Packaging Of Drug Delivery Absorbable Adjuncts,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,858, “Drug Administration Devices That Communicate With Surgical Hubs,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,859, “Controlling Operation Of Drug Administration Devices Using Surgical Hubs,” filed October 13, 2020; U.S. Patent Application No. 17 / 068,863, “Patient Monitoring Using Drug Administration Devices,” filed October 13, 2020; and “Monitoring And Communicating Information Using Drug Administration Further details are provided in U.S. Patent Application No. 17 / 068,865, entitled “Devices,” and U.S. Patent Application No. 17 / 068,867, entitled “Aggregating And Analyzing Drug Administration Data,” filed on 13 October 2020, which are incorporated herein by reference in their entirety.

[0108] The surgical hub can be configured to provide visualization of received data on a display within the surgical setting, so that the physician in the surgical setting can view the data and thereby gain an understanding of the operation of the imaging devices used in the surgical setting. Such information provided through visualization may include text and / or images.

[0109] Figure 19 illustrates one embodiment of a surgical system 802, which includes a surgical hub 806 (e.g., surgical hub 706 in Figure 18 or other surgical hubs described herein), a robotic surgical system 810 (e.g., robotic surgical procedure system 110 in Figure 1 or other robotic surgical systems described herein), and a visualization system 808 (e.g., visualization system 100 in Figure 1 or other visualization systems described herein). The surgical hub 806 can communicate with a cloud as discussed herein. Figure 19 shows the surgical system 802 used to perform a surgical procedure on a patient lying on an operating table 814 in an operating room 816. The robotic system 810 includes a surgeon's console 818, a patient-side cart 820 (surgical robot), and a surgical hub 822 of the robotic system. The surgical hub 822 of the robotic system is generally configured similarly to the surgical hub 822 and can communicate with a cloud. In some embodiments, the surgical hub 822 and surgical hub 806 of the robotic system can be combined. The patient-side cart 820 allows a physician, such as a surgeon, nurse, and / or other physician, to operate an intelligent surgical tool 812 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 818. Images of the surgical site can be acquired by an imaging device 824 (e.g., imaging device 120 in Figure 1, or other imaging devices described herein), which can be operated by the patient-side cart 820 to change the orientation of the imaging device 824. Images of the surgical site can be processed using the surgical hub 822 of the robotic system and then displayed to the surgeon through the surgeon's console 818.

[0110] The main display 819 is positioned in the sterile field of the operating room 816 and is configured to be visible to the operator at the operating table 814. In addition, in this illustrated embodiment, a visualization tower 818 may be positioned outside the sterile field. The visualization tower 818 includes a first non-sterile display 807 and a second non-sterile display 809, facing opposite directions from each other. The visualization system 808, guided by the surgical hub 806, is configured to utilize displays 807, 809, and 819 to coordinate the flow of information to physicians inside and outside the sterile field. For example, the surgical hub 806 can cause the visualization system 808 to display snapshots and / or videos of the surgical site, such as those acquired by the imaging device 824, on one or both of the non-sterile displays 807 and 809, while maintaining live video of the surgical site on the main display 819. The snapshots and / or videos on the non-sterile displays 807 and / or 809 can, for example, allow a non-sterile physician to perform diagnostic steps related to the surgical procedure.

[0111] The surgical hub 806 is configured to send diagnostic input or feedback entered by a non-sterile physician in the visualization tower 818 to the main display 819 in the sterile field, which can then be viewed by a sterile physician at the operating table 814. For example, the input can take the form of a modified snapshot and / or video displayed on the non-sterile display 807 or 809, which can then be sent to the main display 819 by the surgical hub 806.

[0112] The surgical hub 806 is configured to adapt the information flow to the display of the intelligent surgical instrument 812, as described in various U.S. patent applications incorporated herein by reference. Diagnostic inputs or feedback entered by a non-sterile operator in the visualization tower 818 can be sent by the surgical hub 806 to a display 819 in the sterile field, which can be viewed by the operator of the surgical instrument 812 and / or the physician in the sterile field.

[0113] The intelligent surgical instruments 812 and the imaging device 824, which is also an intelligent surgical tool, are used with the patient in surgical procedures as part of the surgical system 802. Other intelligent surgical instruments 812a that can be used in surgical procedures, for example, which are detachably coupled to the patient-side cart 820 and can communicate with the robotic surgical system 810 and the surgical hub 806, are also shown in Figure 19 as being available. Non-intelligent (or "dammed") surgical instruments 817, such as scissors, trocars, cannulas, and scalpels that cannot communicate with the robotic surgical system 810 and the surgical hub 806, are also shown in Figure 19 as being available for use.

[0114] Operation of intelligent surgical instruments An intelligent surgical device may store algorithms configured to be executable onboard by the intelligent surgical device, for example, its processor, in its memory, in order to control the operation of the intelligent surgical device. In some embodiments, instead of, or in addition to, being stored on the intelligent surgical device, the algorithms may be stored on a surgical hub, for example, its memory, which is configured to communicate with the intelligent surgical device.

[0115] The algorithm is stored in the form of one or more sets of data points that define and / or represent commands, notifications, signals, etc., to control the functions of the intelligent surgical device. In some embodiments, data collected by the intelligent surgical device can be used by the intelligent surgical device, for example, by the processor of the intelligent surgical device, to modify at least one variable parameter of the algorithm. As discussed above, a surgical hub can communicate with the intelligent surgical device, and therefore data collected by the intelligent surgical device can be communicated to the surgical hub, and / or data collected by another device communicating with the surgical hub can be communicated to the surgical hub, and data can be communicated from the surgical hub to the intelligent surgical device. Thus, instead of, or in addition to, the intelligent surgical device being configured to modify stored variable parameters, the surgical hub can be configured to communicate at least one modified variable to the intelligent surgical device, either alone or as part of an algorithm, and / or the surgical hub can communicate commands to the intelligent surgical device and modify at least one variable as determined by the surgical hub.

[0116] At least one variable parameter lies between the algorithm's data points, for example, included in the instructions for operating the intelligent surgical device, and each can therefore be modified by changing one or more of the algorithm's stored data points. After at least one variable parameter is modified, subsequent execution of the algorithm follows the modified algorithm. Thus, the operation of the intelligent surgical device over time can be managed for the patient to increase the beneficial use of the intelligent surgical device by considering the patient's actual situation and the actual conditions and / or outcomes of the surgical procedure in which the intelligent surgical device is used. Modifying at least one variable parameter is automated to improve patient outcomes. Therefore, the intelligent surgical device can be configured to provide personalized medicine based on the patient and the patient's surrounding conditions in order to provide a smart system. In a surgical setting in which the intelligent surgical device is used during the execution of a surgical procedure, the automatic modification of at least one variable parameter may allow the intelligent surgical device to be controlled based on data collected during the execution of the surgical procedure, which may help ensure that the intelligent surgical device is used efficiently and accurately and / or help reduce the possibility of harming the patient by damaging important anatomical structures.

[0117] At least one variable parameter can be one of a variety of different operating parameters. Examples of variable parameters include motor speed, motor torque, energy level, energy application duration, tissue compression rate, jaw closure rate, cutting element speed, and load threshold.

[0118] Figure 20 illustrates one embodiment of an intelligent surgical instrument 900, which includes a memory 902 storing an algorithm 904 having at least one variable parameter. The algorithm 904 may be a single algorithm, or it may include multiple algorithms, for example, separate algorithms for different modes of operation of the surgical instrument, each algorithm having at least one variable parameter. The intelligent surgical instrument 900 may be the surgical device 102 in Figure 1, the imaging device 120 in Figure 1, the surgical device 202 in Figure 8, the imaging device 220 in Figure 8, the surgical device 402 in Figure 15, the surgical device 502a in Figure 17, the surgical device 502b in Figure 17, the surgical device 712 in Figure 18, the surgical device 812 in Figure 19, the imaging device 824 in Figure 19, or other intelligent surgical instruments. The surgical instrument 900 also includes a processor 906 configured to execute the algorithm 904 and control the operation of at least one mode of the surgical instrument 900. To execute algorithm 904, processor 906 is configured to execute a program stored in memory 902 to access multiple data points of algorithm 904 in memory 902.

[0119] The surgical instrument 900 also includes a communication interface 908, such as a wireless transceiver or other wired or wireless communication interface, configured to communicate with another device, such as a surgical hub 910. The communication interface 908 can be configured to enable one-way communication, such as providing data to a remote server (e.g., a cloud server or other server) and / or a local surgical hub server, and / or receiving instructions or commands from the remote server and / or the local surgical hub server, or two-way communication, such as providing information, messages, data, etc., about the surgical instrument 900 and / or data stored therein, and receiving instructions from a physician, a remote server regarding software updates, a local surgical hub server regarding software updates, etc.

[0120] The surgical instrument 900, simplified in Figure 20, may include additional components such as a bus system, a handle, an elongated shaft having an end effector at its distal end, and a power supply. The processor 906 can also be configured to execute instructions stored in memory 902 to control the device 900, including other electrical components such as a communication interface 908, an audio speaker, and a user interface.

[0121] The processor 906 is configured to modify at least one variable parameter of algorithm 904 so that subsequent executions of algorithm 904 conform to the modified at least one variable parameter. To modify at least one variable parameter of algorithm 904, the processor 906 is configured to modify or update the data point of at least one variable parameter in memory 902. The processor 906 can be configured to modify at least one variable parameter of algorithm 904 in real time during the execution of a surgical procedure, in conjunction with the use of the surgical device 900, which can address real-time conditions.

[0122] In addition to or instead of the processor 906 modifying at least one variable parameter, the processor 906 may be configured to modify algorithm 904 and / or at least one variable parameter of algorithm 904 in response to instructions received from the surgical hub 910. In some embodiments, the processor 906 is configured to modify at least one variable parameter only after communicating with and receiving instructions from the surgical hub 910, which may help ensure coordinated action of the surgical instrument 900 with other aspects of the surgical procedure in which the surgical instrument 900 is used.

[0123] In an exemplary embodiment, the processor 906 executes algorithm 904 to control the operation of the surgical instrument 900, modifies at least one variable parameter of algorithm 904 based on real-time data, and executes algorithm 904 to control the operation of the surgical instrument 900 after modifying at least one variable parameter.

[0124] Figure 21 illustrates one embodiment of a method 912 for using a surgical instrument 900, which includes changing at least one variable parameter of algorithm 904. A processor 906 controls the operation of the surgical instrument 900 by executing algorithm 904 stored in memory 902 (914). Based on either subsequently known data and / or subsequently collected data, the processor 904 changes at least one variable parameter of algorithm 904 as discussed above (916). After changing at least one variable parameter, the processor 906 controls the operation of the surgical instrument 900 by executing algorithm 904 using the changed at least one variable parameter (918). The processor 904 may change at least one variable parameter any number of times, for example, 0, 1, 2, 3 times, etc., during the execution of a surgical procedure (916). In any part of method 912, the surgical instrument 900 may communicate with one or more computer systems, such as a surgical hub 910, a cloud server, or other remote servers, using the communication interface 908 to provide data thereto and / or receive instructions therefrom.

[0125] Situational awareness The behavior of intelligent surgical instruments can be altered based on the patient's situational awareness. The behavior of intelligent surgical instruments can also be manually altered by the user of the intelligent surgical instrument handling the instrument differently, providing different inputs to the instrument, or stopping the use of the instrument. Additionally or alternatively, the behavior of intelligent surgical instruments can be automatically altered by the instrument's algorithm, for example, by changing at least one variable parameter of the algorithm. As described above, the algorithm can be automatically adjusted without user input requesting the change. Automating adjustments during the execution of a surgical procedure can help save time, allow the physician to focus on other aspects of the surgical procedure, and / or facilitate the process of using the surgical instrument for the physician, each of which can improve patient outcomes by avoiding critical structures, controlling the surgical instrument considering the types of tissues being used and / or nearby, etc.

[0126] The visualization systems described herein can be used as part of a context-aware system that may be embodied or performed by a surgical hub, e.g., surgical hub 706, surgical hub 806, or other surgical hubs described herein. In particular, characterizing, identifying, and / or visualizing surgical instruments (including their position, orientation, and actions), tissues, structures, users, and / or other things located in the surgical field or operating room can be used by the context-aware system to provide contextual data that allows inferences to be made about the type of surgical procedure or process being performed, the type of tissue and / or structure being manipulated by the surgeon or other physician, and other information. This contextual data can then be used by the context-aware system to provide alerts to the user, suggest subsequent processes or actions to be undertaken by the user, prepare surgical devices in anticipation of their use (e.g., activating an electrosurgical generator in anticipation of the use of an electrosurgical instrument in a subsequent process of the surgical procedure), and control the operation of intelligent surgical instruments (e.g., customizing the operating parameters of an algorithm for a surgical instrument, as will be discussed further below).

[0127] An "intelligent" device that includes a control algorithm that responds to sensed data can be an improvement over a "data-dumb" device that operates without considering sensed data, for example, by having at least one variable parameter of a modified algorithm. However, some sensed data may be incomplete or inconclusive when considered in isolation, without context, for example, the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the procedure context (e.g., the type of tissue being operated on or the type of procedure being performed), a control algorithm may inaccurately or suboptimally control the surgical device given sensed data without specific context. For example, the optimal form of a control algorithm for controlling a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different types of tissue have different properties (e.g., resistance to tearing, ease of cutting, etc.) and therefore respond differently to actions taken by the surgical instrument. Thus, even when the same measurement is sensed for a particular parameter, it may be desirable for the surgical instrument to take different actions. As one embodiment, the optimal mode of control for a surgical stapler in response to an unexpectedly high force being detected to close its end effector varies depending on whether the type of tissue is susceptible to tearing or resistant to tearing. For tear-prone tissues, such as lung tissue, the instrument's control algorithm optimally decelerates the motor in response to the unexpectedly high force required to close in order to avoid tearing the tissue, for example by changing a variable parameter that controls the motor speed or torque to slow down the motor. For tear-resistant tissues, such as stomach tissue, the instrument's control algorithm optimally accelerates the motor in response to the unexpectedly high force required to close in order to ensure that the end effector is properly clamped to the tissue, for example by changing a variable parameter that controls the motor speed or torque to speed up the motor.Without knowing whether lung or stomach tissue has been clamped, the algorithm may be modified to a suboptimal state or remain unchanged at all.

[0128] A surgical hub can be configured to derive information about the surgical procedure being performed based on data received from various data sources, and therefore to control modular devices accordingly. In other words, a surgical hub can be configured to infer information about the surgical procedure from received data, and then, based on the inferred context of the surgical procedure, to control modular devices operably coupled to the surgical hub. Modular devices can include any surgical device controllable by a context-aware system, such as a visualization system device (e.g., a camera or display screen) or a smart surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, a surgical stapler, a smoke exhauster, etc.). Modular devices can include sensors configured to detect parameters associated with the patient on whom the device is being used, and / or parameters associated with the modular device itself.

[0129] Contextual information derived or inferred from received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon (or other physician), the type of tissue being operated on, or the body cavity being targeted by the surgical procedure. The surgical hub context-aware system can be configured to derive contextual information from data received from data sources in various different ways. In an exemplary embodiment, the contextual information received by the surgical hub context-aware system is associated with a specific control modulus, or set of control modulus, of one or more modular devices. Each control modulus corresponds to a variable parameter. In one embodiment, the context-aware system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from a database, patient monitoring devices, and / or modular devices) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another embodiment, the situation-aware system may include a lookup table that stores pre-characterized contextual information relating to a surgical procedure, associated with one or more inputs (or ranges of inputs) corresponding to that contextual information. In response to a query with one or more inputs, the lookup table may return the corresponding contextual information of the situation-aware system to control at least one modular device. In another embodiment, the situation-aware system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices when contextual information is provided as input.

[0130] A surgical hub incorporating a context-aware system can offer numerous advantages to the surgical system. One advantage is improved interpretation of sensed and collected data, which improves processing accuracy and / or data utilization during the surgical procedure. Another advantage is that a context-aware system for the surgical hub can improve surgical outcomes by adjusting surgical instruments (and other modular devices) for the specific context of each surgical procedure (e.g., for different tissue types) and verifying actions during the procedure. Yet another advantage is that the context-aware system can improve physician efficiency when performing surgical procedures by automatically suggesting the next steps, providing data, and adjusting displays and other modular devices in the operating room according to the specific context of the procedure. Another advantage is that the context-aware surgical hub can proactively and automatically control modular devices according to the specific step of the surgical procedure being performed, such as by proactively activating a generator to which an RF electrosurgical instrument is connected if it determines that a subsequent step of the procedure requires the use of an instrument, thereby reducing the number of times the physician needs to interact with or control the surgical system during the surgical procedure. By actively activating the energy source, the device can be ready for use as soon as the preceding steps of the procedure are completed.

[0131] For example, a situation-aware surgical hub can be configured to determine what type of tissue is being operated on. Therefore, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the situation-aware surgical hub can be configured to precisely accelerate or decelerate the motor of the surgical instrument for that type of tissue, for example, by changing or triggering a change in at least one variable parameter of an algorithm for the surgical instrument relating to motor speed or torque.

[0132] In another embodiment, the type of tissue being operated on may influence the adjustments made to the compression rate and load threshold of the surgical stapler for specific interstitial space measurements. A context-aware surgical hub can be configured to infer whether the surgical procedure being performed is a thoracic or abdominal procedure, and the surgical hub can determine whether the tissue clamped by the end effector of the surgical stapler is lung tissue (in a thoracic procedure) or gastric tissue (in an abdominal procedure). The surgical hub can then be configured to cause adjustments to the compression rate and load threshold of the surgical stapler to be appropriate for that type of tissue, for example, by changing or causing a change in at least one variable parameter of the algorithm for the surgical stapler relating to the compression rate and load threshold.

[0133] In yet another embodiment, the type of body cavity being operated on during an insufflation procedure may affect the function of the smoke exhauster. The situation-aware surgical hub can be configured to determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and to determine the type of procedure. Since certain types of procedures are generally performed in specific body cavities, the surgical hub can be configured to control the motor speed of the smoke exhauster appropriately for the body cavity being operated on, for example, by changing or causing a change in at least one variable parameter of the algorithm for the smoke exhauster relating to the motor speed. Thus, the situation-aware surgical hub can provide a consistent amount of smoke exhaust for both thoracic and abdominal procedures.

[0134] In yet another embodiment, the type of procedure being performed may affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. A context-aware surgical hub can be configured to determine whether the surgical procedure is an arthroscopy. The surgical hub can be configured to compensate for the fluid-filled environment by adjusting the RF power level or ultrasonic amplitude of the generator (e.g., adjusting the energy level) by changing or triggering a change in at least one variable parameter of the algorithm for the instrument and / or generator relating to the energy level. Relatedly, the type of tissue being operated on may affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The context-aware surgical hub can be configured to determine what type of surgical procedure is being performed and then customize the energy levels for the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure by changing or triggering a change in at least one variable parameter of the algorithm for the instrument and / or generator relating to the energy level. Furthermore, the situation-aware surgical hub can be configured to adjust the energy levels of ultrasonic surgical instruments or RF electrosurgical instruments not simply for each surgery, but throughout the course of the surgical procedure. The situation-aware surgical hub can be configured to determine which stage of the surgical procedure is being performed or is continuing, and then update the control algorithms of the generator and / or ultrasonic surgical instruments or RF electrosurgical instruments to set the energy levels to values ​​appropriate for the expected tissue type according to the stage of the surgical procedure.

[0135] In another embodiment, the situational awareness surgical hub may be configured to determine whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display, according to the characteristics of the surgical site that the surgeon and / or physician are expected to need to see. The surgical hub may be configured to proactively change the displayed view (supplied, for example, from an imaging device for a visualization system) as needed, so that the display automatically adjusts throughout the surgical procedure.

[0136] In yet another embodiment, the situation-aware surgical hub can be configured to determine which steps of a surgical procedure are being performed or will be performed, and whether specific data or comparisons of data are necessary for that step of the surgical procedure. The surgical hub can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon or physician to ask for specific information.

[0137] In another embodiment, a situation-aware surgical hub can be configured to determine whether a surgeon and / or other physician is making an error or deviating from a different expected sequence of actions during the course of a surgical procedure, for example, as provided in a preoperative surgical plan. For example, the surgical hub can be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the course of the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub has determined is being performed. The surgical hub can be configured to provide (visual, auditory, and / or tactile) alerts indicating that an unexpected action is being taken or an unexpected device is being used at a particular step in the surgical procedure.

[0138] In certain examples, the operation of a robotic surgical system, such as one of the various robotic surgical systems described herein, can be controlled by a surgical hub based on its situational awareness and / or feedback from its components, and / or information from the cloud (e.g., cloud 713 in Figure 18).

[0139] Embodiments of the situation recognition system and embodiments of using the situation recognition system during the execution of surgical procedures are described in U.S. Patent Application No. 16 / 729,772, “Analyzing Surgical Trends By A Surgical System,” filed on December 30, 2019; U.S. Patent Application No. 16 / 729,747, “Dynamic Surgical Visualization Systems,” filed on December 30, 2019; U.S. Patent Application No. 16 / 729,744, “Visualization Systems Using Structured Light,” filed on December 30, 2019; U.S. Patent Application No. 16 / 729,778, “System And Method For Determining, Adjusting, And Managing Resection Margin About A Subject Tissue,” filed on December 30, 2019; and “Surgical Systems For Proposing And Corroborating Organ Portion U.S. Patent Application No. 16 / 729,729, entitled "Removals", U.S. Patent Application No. 16 / 729,778, entitled "Surgical System For Overlaying Surgical Instrument Data Onto A Virtual Three Dimensional Construct Of An Organ", filed on December 30, 2019, U.S. Patent Application No. 16 / 729,751, entitled "Surgical Systems For Generating Three Dimensional Constructs Of Anatomical Organs And Coupling Identified Anatomical Structures Thereto", filed on December 30, 2019, U.S. Patent Application No. 16 / 729,729, entitled "Surgical Systems Correlating Visualization Data And Powered Surgical Instrument Data", filed on December 30, 2019Further details are provided in U.S. Patent Application No. 16 / 729,737, entitled "Adaptive Surgical System Control According To Surgical Smoke Cloud Characteristics," filed December 30, 2019; U.S. Patent Application No. 16 / 729,796, entitled "Adaptive Surgical System Control According To Surgical Smoke Particulate Characteristics," filed December 30, 2019; U.S. Patent Application No. 16 / 729,803, entitled "Adaptive Visualization By A Surgical System," filed December 30, 2019; and U.S. Patent Application No. 16 / 729,807, entitled "Method Of Using Imaging Devices In Surgery," filed December 30, 2019.

[0140] Lung surgery Various embodiments of the devices, systems, and methods described herein may relate to surgical procedures performed on the lungs. For example, lung resection (e.g., lobectomy) is a surgical procedure in which all or part of a lung lobe, for example, one or more lobes, are removed. The purpose of lung resection is, for example, to treat a lung that is damaged or affected as a result of lung cancer, emphysema, or bronchiectasis.

[0141] During lung resection, the lung is first deflated, and then one or more incisions are made on the patient's side, between the patient's ribs, to access the lung laparoscopically. Surgical instruments such as grippers and laparoscopes are inserted through the incisions. Once the infected or damaged area of ​​the lung is identified, that area is resected from the lung and removed through one or more incisions. The resected area and one or more incisions can be closed, for example, by surgical staples or sutures.

[0142] Since the lungs contract during surgery, the lungs or specific parts of the lungs may need to be mobilized to allow surgical instruments to reach the surgical site. This mobilization can be performed by grasping the outer tissue layers of the lung with a grasper and applying force to the lung through the grasper. However, the pleura and parenchymal tissues of the lung are very fragile and can therefore be easily torn or ruptured under applied force. Additionally, during mobilization, the grasper may cut off the blood supply to one or more areas of the lung.

[0143] Furthermore, a breathing tube can be placed in the patient's airway, allowing each lung to be inflated individually during surgery. Lung inflation can allow the lung to be moved, matched to preoperative imaging, and / or the surgeon to check for leakage in the incised area. However, inflating the entire lung results in a loss of working space around the lung due to filling the pleural cavity. Additionally, inflating the entire lung can be time-consuming and does not guarantee easy leakage detection if multiple parts of the lung are operated on during the surgical procedure.

[0144] Surgical procedure for the colon Various aspects of the devices, systems, and methods described herein may relate to surgical procedures performed on the colon. For example, surgery is the primary treatment for early-stage colon cancer. The type of surgery used depends on the stage (severity) of the cancer and the goals of the surgery, provided the cancer is located within the colon. Some early-stage colon cancers (stage 0 and some early stage I tumors) and most polyps can be removed during a colonoscopy. However, if the cancer has progressed, local resection or colectomy may be necessary. Colectomy is surgery to remove all or part of the colon. In certain cases, nearby lymph nodes are also removed. When only a portion of the colon is removed, it is called hemicolectomy, partial colectomy, or segmental resection, and the surgeon removes the affected portion of the colon along with a small segment of the unaffected colon on either side. Typically, about one-quarter to one-third of the colon is removed, depending on the size and location of the cancer. Major resections of the large intestine are illustrated in Figure 21A, where AB is a right hemicolectomy, AC is a dilated right hemicolectomy, BC is a transverse colectomy, CE is a left hemicolectomy, DE is a sigmoid colectomy, DF is an anterior resection, DG is a (very) low anterior resection, DH is an abdominoperineal resection, AD is a subtotal colectomy, AE is a total colectomy, and AH is a total rectocolectomy. Once the resection is complete, the remaining intact section of the colon is reattached.

[0145] Colectomy can be performed through open colectomy, in which a single incision through the abdominal wall is used to access the colon for the separation and removal of the affected colonic tissue, and through laparoscopic-assisted colectomy. In laparoscopic-assisted colectomy, surgery is performed through many smaller incisions using surgical instruments, and a laparoscope is used to remove the entire colon or a portion thereof through the small incisions. At the start of the procedure, the abdomen is inflated with gas, such as carbon dioxide, to provide working space for the surgeon. The laparoscope transmits images of the abdominal cavity, giving the surgeon a magnified view of the patient's internal organs on a monitor or other display. Several other cannulas are inserted to allow the surgeon to work inside the colon and remove a portion of it. Once the affected portion of the colon has been removed, the remaining ends of the colon are joined together, for example, by staples or sutures. The entire procedure can be completed through the cannulas or by extending one of the small cannula incisions.

[0146] Obtaining a proper surgical field during laparoscopic-assisted colectomy is often difficult. Because the incision is frequently made deep within the pelvis, it is often challenging to adequately visualize its area. As a result, the lower rectum must be lifted and rotated to gain access to the veins and arteries around both sides of the rectum during mobilization. Tissue bunching and / or hyperstretching may occur during manipulation of the lower rectum. Additionally, tumors within the rectum can cause adhesions in the surrounding pelvis, which may necessitate releasing the rectal stump and mobilizing the mesentery and blood supply before transection and removal of the tumor.

[0147] Furthermore, multiple grippers are required to position the tumor for removal from the colon. During the incision of the colon, the tumor should be positioned under tension, which requires grasping and stretching the healthy tissue surrounding the colon. However, manipulating the tissue surrounding the tumor can result in reduced blood flow and trauma due to the high gripping force applied by the grippers to the tissue. Additionally, during colectomy, the transverse and upper descending colon may need to be made mobile to allow the healthy remaining colon to be lowered to connect to the rectal stump after the section of the colon containing the tumor has been severed and removed.

[0148] After a colectomy, the remaining healthy portions of the colon must be reattached to each other to create a pathway for waste products to leave the body. However, when performing a colectomy using laparoscopic instruments, a single entry port may not have a sufficient range of motion to move one end of the colon to the connecting portion. Therefore, a second entry port is needed to allow for the laparoscopic insertion of surgical instruments to help mobilize the colon in order to properly position it.

[0149] Stomach surgery procedure Various aspects of the devices, systems, and methods described herein may relate to surgical procedures performed on the stomach. For example, surgery is the most common treatment for gastric cancer. When surgery is required for gastric cancer, the goal is to remove the entire tumor as well as a good margin of healthy gastric tissue surrounding the tumor. Gastric cancer can be removed using different procedures. The type of procedure used depends on where in the stomach the cancer is located and how much of a nearby area the cancer has grown. For example, endoscopic mucosal resection (EMR) and endoscopic submucosal dissection (ESD) procedures on the stomach can be used to treat some early-stage cancers. These procedures do not require cutting the skin; instead, the surgeon inserts an endoscope through the patient's throat into the stomach. Then, a surgical tool (e.g., MEGADYNE® Tissue Dissector or Electrosurgical Pencil) is passed through the working channel of the endoscope to remove the tumor and several layers of normal gastric wall beneath and around it.

[0150] Other surgical procedures performed on the stomach include subtotal (partial) or total gastrectomy, which can be performed as an open procedure. For example, surgical instruments are inserted through a large incision in the abdominal skin, or, as a laparoscopic procedure, surgical instruments are inserted into the abdomen through several small incisions. For example, laparoscopic gastrectomy generally involves infusing carbon dioxide gas into the abdominal cavity to a pressure of approximately 15 mmHg of mercury. The abdominal wall is perforated, and then a straight tubular cannula or trocar, such as a cannula or trocar with a diameter ranging from approximately 5 mm to 10 mm, is inserted into the abdominal cavity. A laparoscope connected to a monitor in the operating room is used to visualize the surgical field and is positioned through one of the trocars. Laparoscopic surgical instruments are positioned through two or more additional cannulas or trocars for manipulation by medical personnel, such as a surgeon and surgical assistant, to remove the desired portion of the stomach.

[0151] In certain cases, a coordinated laparoscopic and endoscopic surgical procedure can be used to remove gastric tumors. This coordinated procedure typically involves the introduction of an endoscope, such as a gastroscopy, and a laparoscopic trocar. The laparoscope and tissue manipulation and dissection instruments are introduced through the trocar. The tumor location can be identified via the endoscope, and a cutting element inserted into the working channel of the endoscope is then used for submucosal resection around the tumor. Next, laparoscopic dissection instruments are used for serosamuscular dissection adjacent to the tumor margin to create an incision through the stomach wall. The tumor is then rotated through this incision from the intraluminal space, e.g., inside the stomach, to the extraluminal space, e.g., outside the stomach. Then, laparoscopic surgical instruments, such as an endocutter, can be used to complete the detachment of the tumor from the stomach wall and to seal the incision.

[0152] Surgical procedure for the intestines Various aspects of the devices, systems, and methods described herein may relate to surgical procedures performed on the intestines. For example, duodenal mucosal resurfacing (DMR) procedures can be performed endoscopically to treat insulin-resistant metabolic diseases such as type 2 diabetes. DMR procedures can be an effective treatment because they affect the detection of food. DMR procedures disrupt duodenal function so that food tends to be sensed deeper in the intestines than normal, for example, after passing through the duodenum (the first part of the small intestine). Thus, the patient's body senses sugar deeper in the intestines than typical, and therefore reacts to sugar more slowly than typical, thus improving blood glucose control. Irregular duodenal function alters the body's typical response to food, and via nervous and chemical signals, causes the body to adapt its response to glucose levels and increase insulin levels.

[0153] In DMR procedures, the duodenal mucosa is lifted using saline solution or similar, and then ablated using an ablation device advanced into the duodenum, for example, through the working channel of an endoscope. Lifting the mucosa before ablation helps protect the outer layer of the duodenum from damage caused by ablation. After the mucosa is ablated, it regenerates later. An example of an ablation device is the NeuWave® ablation probe (available from Ethicon US LLC of Cincinnati, OH). Another example of an ablation device is the Hyblate catheter ablation probe (available from Hyblate Medical of Misgav, Israel). Another example of an ablation device is the Barxx® HaloFlex (available from Medtronic of Minneapolis, MN).

[0154] Figure 21B illustrates one embodiment of a DMR procedure. As shown in Figure 21B, the laparoscope 1400 is positioned outside the duodenum 1402 for external visualization of the duodenum 1402. The endoscope 1404 is advanced orally through the esophagus 1406 and stomach 1408 into the duodenum 1402 for internal visualization of the duodenum 1402. The ablation device 1410 is advanced through the working channel of the endoscope 1404 and extends distally from the endoscope 1404 into the duodenum 1402. The balloon 1412 of the ablation device 1410 is shown expanded or inflated in Figure 21B. The expanded or inflated balloon 1412 can help center the electrode of the ablation device so that even circumferential ablation can occur before the ablation device 1410 is advanced and / or retracted to repeat the ablation. Before the mucosa is ablated using the ablation device 1410, the duodenal mucosa is lifted with saline solution or the like. In some embodiments, in addition to or instead of including a balloon 1412, the ablation device 1410 can be made inflatable / collapsed by using an electrode array or basket configured to expand and collapse.

[0155] External laparoscopic visualization of the duodenum 1402 can enable thermal monitoring of the duodenum 1402, which can help ensure that the outer layer of the duodenum 1402 is not damaged by ablation of the duodenal mucosa, such as when the duodenum is perforated. Various embodiments of thermal monitoring are discussed, for example, below and further in U.S. Patent Application No. 63 / 249,658, entitled "Surgical Devices, Systems, and Methods For Control Of One Visualization With Another," filed September 29, 2021. The endoscope 1404 and / or ablation device 1410 may include reference markers that can be configured for laparoscopy 1400 to visualize through duodenal tissue, for example, by using invisible light, which can help the laparoscopy 1400 determine where the duodenum 1402 should be visualized externally at the site where ablation occurs. Various embodiments of the reference marker are discussed further, for example, in U.S. Patent Application No. 63 / 249,652, entitled "Surgical Devices, Systems, Methods Using Fiducial Identification And Tracking," filed September 29, 2021, and U.S. Patent Application No. 63 / 249,658, entitled "Surgical Devices, Systems, And Methods For Control Of One Visualization With Another," filed September 29, 2021.

[0156] Coordinated control of surgical instruments In various embodiments, the Disclosure provides methods, devices, and systems for controlling coordinated surgical instruments. For example, in one embodiment, the system may include a first surgical instrument inserted into a first portion of a patient's body cavity and configured to operate on a first surgical treatment site within the body cavity, and a second surgical instrument inserted into a second portion of the body cavity and configured to operate on a second surgical treatment site within the body cavity. The system may also include a first endoscope and a second endoscope. The first endoscope may have a first image sensor that can be positioned within the first portion of the body cavity such that the second surgical instrument is not within the field of view of the first image sensor. The second endoscope may also have a second image sensor that can be positioned within the second portion of the body cavity such that the first surgical instrument is not within the field of view of the second image sensor. A controller may be included in the system and configured to receive the acquired first and second images, determine the first and second locations of the first surgical instrument, determine the distance and orientation of the first surgical instrument relative to the second surgical instrument, and, based on the determined distance and orientation, trigger a movement of at least one of the first and second surgical instruments within the body cavity. By triggering a movement based on the determined distance and orientation, the controller may help the user simplify the movement of the two instruments when the user cannot directly see or visualize where the two instruments are relative to each other due to the obscured view of the two endoscopes. By simplifying the movement between the two instruments, the controller may help protect the patient's health and make surgical procedures more efficient.

[0157] For example, Figure 22 provides a schematic diagram of one exemplary surgical system 1000 that can provide coordinated control of surgical instruments with respect to the location and movement of various instruments, such as the first and second surgical instruments discussed above. As shown, the system 1000 includes a first surgical instrument 1010, a second surgical instrument 1030, a first endoscope 1020, a second endoscope 1040, and a controller 1050.

[0158] The first surgical instrument 1010 and the second surgical instrument 1030 can each be any suitable surgical device configured to manipulate and / or treat tissue. The first surgical instrument 1010 and the second surgical instrument 1030 can each be similar to the surgical device 102 in Figure 1, the surgical device 202 in Figure 8, or other surgical devices described herein. As described above, examples of surgical devices include surgical dissection instruments, surgical staplers, surgical grippers, clip applicators, smoke extractors, surgical energy devices (e.g., unipolar probes, bipolar probes, ablation probes, ultrasound devices, ultrasound end effectors, etc.). For example, in some embodiments, the first surgical instrument 1010 and / or the second surgical instrument 1030 may include an end effector having opposing jaws that extend from the distal end of the shaft of the surgical device and are configured to engage tissue between them.

[0159] The first endoscope 1020 and the second endoscope 1040 may each include an imaging device configured to acquire images of the surgical site in minimally invasive surgical procedures, including various flexible endoscope systems with image sensors, as discussed above. The first endoscope 1020 and the second endoscope 1040 may each be similar to the imaging device 120 in Figure 1, the imaging device 220 in Figure 8, or other imaging devices described herein. Although some implementations of this subject matter are described herein as using one or more endoscopes to acquire images of the surgical site, any type of scope suitable for use in minimally invasive surgical procedures may be used in conjunction with the systems, methods, and devices described herein. As described above, examples of scopes include arthroscopes, angioscopes, bronchoscopes, common cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinal scopes, double-balloon enteroscopes, esophagogastroduodenoscopies (gastroscopies), laryngoscopes, nasopharyngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, exoscopes, and self-propelled dual-flex endoscopes. One or more of these exemplary types of scopes can be used together in any feasible combination in minimally invasive surgical procedures.

[0160] The controller 1050 includes a processor 1051 configured to perform one or more of the operations, and a memory 1052 configured to store instructions for the processor 1051 to perform the operations. The controller 1050 also includes a first surgical instrument interface 1053, a first endoscope interface 1054, a second surgical instrument interface 1055, and a second endoscope interface 1056. As shown in Figure 22, the first surgical instrument 1010 is coupled to the controller 1050 via the first surgical instrument interface 1053 and can therefore receive motion and operation instructions from the processor 1051. The first endoscope 1020 is coupled to the controller 1050 via the first endoscope interface 1054 and can therefore provide data characterizing images acquired by the first endoscope 1020 to the processor 1051 for use by the processor 1051 when performing various of the operations, and / or to the memory 1052 for later use by the processor 1051. Similar to the first surgical instrument 1010, the second surgical instrument 1030 is coupled to the controller 1050 via the second surgical instrument interface 1055, and is therefore able to receive motion and operation commands from the processor 1051. Similar to the first endoscope 1020, the second endoscope 1040 is coupled to the controller 1050 via the second endoscope interface 1056, and is therefore able to provide data characterizing the images acquired by the endoscope 1040 to the second processor 1051 for use by the processor 1051 when performing various operations, and / or to memory 1052 for later use by the processor 1051 during operations. In some embodiments, each of the first surgical instrument interface 1053, the first endoscope interface 1054, the second surgical instrument interface 1055, and the second endoscope interface 1056 may differ from one another to correspond to the differences between the controller interfaces of each of the first surgical instrument 1010, the first endoscope 1020, the second surgical instrument 1030, and / or the second endoscope 1040.In some embodiments, the controller 1050 can determine the first and second locations of the first surgical instrument 1010 and the second surgical instrument 1030 relative to each other, and can determine the distance and orientation of the first surgical instrument 1010 relative to the second surgical instrument 1030, as will be discussed further below.

[0161] As shown, system 1000 also includes a display 1060 operably coupled to controller 1050 and configured to graphically depict images acquired by one or more of the first endoscope 1020 and the second endoscope 1040. In the illustrated embodiment, controller 1050 receives streams of image data from each of the first endoscope 1020 and the second endoscope 1040, determines images and / or video feeds from the received image data in real time, and provides the images and / or video feeds to display 1060, on which they are depicted, and for viewing by the user. In some embodiments, controller 1050 can merge the first and second images together to create a merged image or virtual treatment site created from the first and second images for use by the surgeon during the procedure. Additional details regarding various embodiments of the merged image are provided, for example, in U.S. Patent Application No. 63 / 249,980 entitled “Cooperative Access,” filed September 29, 2021, as mentioned above.

[0162] System 1000 can be used in a variety of different surgical procedures involving a variety of different surgical instruments and / or surgical implants. For example, as will be discussed further below, System 1000 can be used to visualize the orientation of implants, control the advancement speed of various instruments, enable synchronized and / or coordinated action between various instruments during various procedures, and enable the determination of various tissue characteristics of surrounding tissues during various procedures.

[0163] Figure 23 shows one embodiment of using System 1000 in a partial jejunal diversion to coordinate the placement of surgical implants, in this illustrated embodiment, the first implant component 1080a and the second implant component 1080b of a two-part magnetic anastomosis device, at a connection or joint surgical treatment site, when it is not possible to visualize each of the surgical instruments 1010, 1030 and the two-part magnetic anastomosis device using a single of the endoscopes 1020, 1040, and when it is not possible to visualize each of the surgical instruments 1010, 1030 and the two-part magnetic anastomosis device. Partial jejunal diversion is used to create a shorter metabolic pathway through the patient's jejunum (small intestine), for example, to alter the rate of GI motility and glucose involvement of digested food. However, the considerable length of a patient's jejunum compared to the limited length of typical endoscopes and surgical instruments presents a challenge to the user when using two separate endoscopes and two separate surgical instruments to position a two-part anastomosis device, as the endoscopes cannot normally or always directly visualize each other or the two-part anastomosis device to help determine the orientation and position of the two-part anastomosis device before deployment. The illustrated embodiment enables determining and controlling the orientation and distance of surgical instruments 1010, 1030, endoscopes 1020, 1040, and implant components 1080a, 1080b as needed to ensure the correct alignment of implant components 1080a, 1080b before deployment. Furthermore, system 1000 can be used in other surgical procedures and with other implants as well.

[0164] As illustrated in Figure 23, the first endoscope 1020 is inserted into the patient's first natural opening, the mouth in this illustrated embodiment, and is maneuvered and advanced through the patient's esophageal sphincter 1091, pyloric sphincter 1092, and duodenojejunal flexure 1094 to the first surgical treatment site 1070a in the patient's jejunum 1090 (or the intermediate portion of the small intestine between the duodenum and ileum). The second endoscope 1040 is inserted into the patient's second natural opening, the rectum in this illustrated embodiment, and is maneuvered and advanced through the ileocecal valve 1093 to the second surgical treatment site 1070b in the jejunum 1090. The first surgical treatment site 1070a and the second surgical treatment site 1070b may be identified before insertion using various external imaging mechanisms such as CT imaging in some embodiments, and sites 1070a and 1070b may be confirmed after insertion of the endoscopes 1020 and 1040 through various external imaging methods such as CT imaging, direct imaging from the endoscopes 1020 and 1040, and the use of various additional instruments such as the laparoscope 1085 shown in Figure 23. In some surgical procedures, the laparoscope 1085 may not be used.

[0165] In the illustrated embodiment, as the first endoscope 1020 and the second endoscope 1040 reach the first surgical treatment site 1070a and the second surgical treatment site 1070b, respectively, each endoscope 1020, 1040 detects its position relative to the other endoscope 1020, 1040, so that both endoscopes 1020, 1040 are positioned and oriented relative to each other at their respective surgical treatment sites 1070a, 1070b, which can help ensure the successful delivery of the first implant component 1080a and the second implant component 1080b at their respective sites 1070a and 1070b. The location and orientation of each endoscope 1020, 1040 are tracked by the controller 1050 of system 1000 using the electromagnetic (EM) tracking tip of each endoscope 1020, 1040 via magnetic field detection to determine the location and orientation of each endoscope 1020, 1040 in the global coordinate system of system 1000, the global coordinate system being known by the controller 1050 communicating with each of the endoscopes 1020, 1040 and each of the surgical instruments 1010, 1030. In the illustrated embodiment, an EM tracking tip is provided, but alternative and / or additional tracking means can be used, such as fiber Bragg gratings, virtual tags, the use of the aforementioned reference markers and probes, identification of known anatomical structures, various 3D scanning techniques such as the use of structured light discussed above, and various sensors and / or imaging systems discussed above. Further details regarding various embodiments for tracking surgical instruments are provided, for example, in U.S. Patent Application No. 63 / 249,980, entitled “Cooperative Access,” filed on September 29, 2021, as mentioned above.

[0166] The first implant component 1080a of the two-part magnetic anastomosis device is releasably attached to the distal end of the first surgical instrument 1010, and the first surgical instrument 1010 with the first implant component 1080a is inserted through the working channel of the first endoscope 1020 until the first implant component 1080a and the most distal portion of the first instrument 1010 are positioned distal to the first endoscope 1020 at the first surgical treatment site 1070a. Similarly, the second implant component 1080b of the two-part magnetic anastomosis device is releasably attached to the distal end of the second surgical instrument 1030, and the second surgical instrument 1030 with the second implant component 1080b is inserted through the working channel of the second endoscope 1040 until the second implant component 1080b and the most distal portion of the second instrument 1030 are positioned distal to the second endoscope 1040 at the second surgical treatment site 1070b. In other embodiments, the first surgical instrument 1010 and / or the second surgical instrument 1030 can be advanced along the outside of the first endoscope 1020 and the second endoscope 1040, respectively, instead of being advanced through the working channel.

[0167] Furthermore, the controller 1050 of system 1000 controls the forward force and speed of the first instrument 1010 and the second instrument 1030, as well as the first endoscope 1020 and the second endoscope 1040, relative to each other and / or to the rendezvous point within the patient. When tracking the first instrument 1010 and the second instrument 1030 and the first endoscope 1020 and the second endoscope 1040, the controller 1050 determines the forward speed, approach vector, applicable force, and / or distance from each other and / or distance from the rendezvous point to define the location within the patient where the first implant component 1080a and the second implant component 1080b are intended to be joined together. As each of the instruments 1010, 1030 and / or endoscopes 1020, 1040 approaches the rendezvous point, the forward speed, applicable force, and / or detection sampling rate are modified by the controller 1050 to ensure that the approaching instruments 1010, 1030 and / or endoscopes 1020, 1040 do not collide with or overshoot the rendezvous point, and to allow for more delicate or precise positioning of the corresponding instruments 1010, 1030 and / or endoscopes 1020, 1040. In some embodiments, the controller 1050 can also display the forward speed, approach vector, applicable force, distance from each other, distance from the rendezvous point (whatever that is intended to mean), and / or detection sampling rate on a display such as the display 1060.

[0168] In the illustrated embodiment of System 1000, Figure 24 shows the forward speed (mm / min) of the first instrument 1010 (identified as the "distal instrument" in Figure 24) and the second instrument 1030 (identified as the "proximal instrument" in Figure 24) through the patient, compared to the distance from the rendezvous point (identified as the "intra-intestinal distance" compared to the "rendezvous point" in Figure 24). The identified rendezvous point illustrated in Figure 24 represents the connected or joined surgical treatment site formed by joining the first treatment site 1070a and the second treatment site 1070b, and the intra-intestinal distance is shown increasing from the proximal end of the patient's intestine to the distal end of the patient's intestine, with the identified rendezvous point approximately in the center. However, in other embodiments and during other procedures, the distance and rendezvous point may represent different biological organs and target sites.

[0169] The controller 1050 also utilizes thresholds and zones or ranges of values ​​to automatically limit the forward speed and distance from the rendezvous point of one or both of the first instrument 1010 and the second instrument 1030 as they pass through the patient, in order to enhance patient safety. The range of the maximum forward speed (similar to the speed limit) and the distance from the treatment sites 1070a, 1070b is selected for the first instrument 1010 and the second instrument 1030, as illustrated in Figure 24. Additionally, in some embodiments, various thresholds and ranges can be correlated. For example, in the embodiment illustrated in Figure 24, a "close to rendezvous point" zone and a speed limit threshold are provided. The "close to rendezvous point" zone represents a pre-selected distance from the rendezvous point and includes a lower distance value 1075a and an upper distance value 1075b on either side of the rendezvous point. Furthermore, the speed limit threshold 1075c represents the upper limit of the speed of the instruments 1010, 1030. However, when the instruments 1010 and 1030 enter the "close to the rendezvous point" zone, the controller 1050 only limits the forward speed of the instruments 1010 and 1030 to the limiting speed threshold 1075c. Before entering the provided range between threshold distance values ​​1075a and 1075b, the forward speed of the instruments 1010 and 1030 is not limited. Figure 25 illustrates multiple distances from the rendezvous point in Figure 24 through the patient's intestine during both proximal and distal approaches. For example, multiple proximal distances P1, P2, P3, P4, P5, P6 associated with a first surgical instrument 1010 and a first endoscope 1030 approaching from a location proximal to the rendezvous point, and multiple distal distances D1, D2, D3, D4, D5, D6 associated with a second surgical instrument 1020 and a second endoscope 1040 approaching from a location distal to the rendezvous point can be identified, and one or more thresholds can be established at one or more of these distances to control the forward speed of the instruments 1010, 1030 and the endoscopes 1020, 1040.

[0170] Taking into account the length and complexity of the intestine, known movement paths can be created for each instrument 1010, 1030 to help the controller 1050 and the user navigate precisely through the patient's intestine to reach the rendezvous point. In such embodiments, points approximately 2 cm apart from each other, such as multiple proximal distances P1, P2, P3, P4, P5, P6 and multiple distal distances D1, D2, D3, D4, D5, D6 illustrated in Figure 25, are picked and tagged during the creation of a computer image along the patient's intestine during enterolateroscopic movement. Thus, the controller 1050 refers to each point P n and D n Knowing the distance to the rendezvous point, as well as the position of each endoscope 1020, 1040 in the global coordinate system, the controller 1050 can track the distance of each endoscope 1020, 1040 to the rendezvous point through the flexible, curved anatomical structure of the patient's intestine. Thus, instead of requiring the user to separately monitor multiple systems, each having its own instrument and endoscope, the controller 1050 can track and limit the forward speed, applicable force, and / or distance from selectable points for the system 1000, including multiple instruments 1010, 1030 and multiple endoscopes 1020, 1040, thereby improving safety and precision during placement, while also allowing the user to focus more fully on the current procedure and less on monitoring and tracking multiple instruments and endoscopes.

[0171] Instruments 1010 and 1030 reach treatment sites 1070a and 1070b, but before the deployment of implant components 1080a and 1080b, instruments 1010 and 1030 interact with each other through at least one shared intact tissue wall, and controller 1050 makes a determination about the tissue based on the interaction. For example, controller 1050 determines tissue properties such as thickness, stiffness, and cross-sectional tissue composition. By determining tissue properties, various specific deployment or treatment sites can be identified based on those specific tissue properties.

[0172] For example, in the embodiment shown in Figure 23, the first surgical treatment site 1070a and the second surgical treatment site 1070b are first identified using various preoperative imaging approaches, such as CT or MRI scans, to determine preliminary locations for implant deployment. However, while manipulating the implant components 1080a and 1080b into place, the controller 1050 monitors tissue characteristics to determine the precise deployment location of each implant component 1080a and 1080b with appropriate tissue characteristics. The determination is made based on one or more different factors, such as measured tissue impedance, tissue thickness, tissue density, cross-sectional tissue composition of the surrounding tissue wall (including mucosa, submucosa, or serosa), multispectral or ultrasound non-visible light spectral imaging, tissue and anatomical visualization, the distance between instruments 1010 and 1030 and endoscopes 1020 and 1040, and resistance from the surrounding tissue to further advancement of instruments 1010 and 1030 and endoscopes 1020 and 1040. In the illustrated embodiment, two endoscopes 1020 and 1040 are aligned with only the wall thickness of each lumen in the intestine separating the endoscopes 1020 and 1040, and thus the distance between the two endoscopes 1020 and 1040 is used to help determine the combined tissue wall thickness between surgical treatment sites 1070a and 1070b. When deploying implant components 1080a and 1080b, the user attempts to identify the location where sufficient compression is demonstrated to cause erosion of the tissue wall between the two implant components 1080a and 1080b, resulting in mating of serosal tissue layers and creating a healing bond between the first surgical treatment site 1070a and the second surgical treatment site 1070b. In other embodiments, different imaging or analytical approaches can be taken to assist the user in visualizing blood flow using non-invasive indocyanine green (ICG), etc., and ensuring that only the intestinal wall is between the two implant components 1080a and 1080b, and that there is no mesentery or connective tissue between them.In some embodiments, the user can further utilize ICG to determine intestinal perfusion viability, particularly at surgical treatment sites 1070a and 1070b. In some embodiments, the user can also use ICG to confirm tissue characteristics or tissue type by comparing tissue characterization performed from within the intestine, such as through the use of optical coherence tomography or confocal lasers, with tissue characterization performed from outside the intestine, such as through the use of multispectral non-contact imaging or impedance contact spectroscopy.

[0173] The two-part magnetic anastomosis device is then deployed to a location that offers the best chance of success based on local tissue characteristics. In some embodiments, the controller 1050 and / or user may request one or more of the different tissue factors discussed above, such as measured tissue impedance, tissue thickness, and tissue density, to help prevent the instruments 1010, 1030 from penetrating the tissue wall within the patient during the surgical procedure, thereby reducing overall damage to the patient. In other embodiments, one or more additional instruments, such as a laparoscope 1085, various probes or lasers, or a balloon, may be used to assist in making additional determination of the surrounding tissue to ensure that only the desired tissue is positioned between the instruments 1010, 1030 and / or to assist in rotating and manipulating the surrounding tissue before implant deployment or other treatment.

[0174] In the exemplary embodiment shown in Figure 23, when the first implant component 1080a and the second implant component 1080b of the two-part magnetic anastomosis device reach their respective treatment sites 1070a and 1070b, the first and second surgical instruments 1010 and 1030 are rotated or articulated as necessary based on the detected location and orientation of each instrument 1010 and 1030 to help ensure the successful delivery of the first implant component 1080a and the second implant component 1080b at their respective sites 1070a and 1070b. Similar to the first endoscope 1020 and the second endoscope 1040 discussed above, the location and orientation of each instrument 1010 and 1030 are tracked by the controller 1050 of system 1000 using an EM tracking chip (and / or other tracking means), and the controller 1050 determines the location and orientation of each instrument 1010 and 1030 in a known global coordinate frame of system 1000.

[0175] While the illustrated embodiment provides implant delivery, the system 1000 can coordinate synchronized or coordinated movements and procedures between various instruments 1010, 1030 and endoscopes 1020, 1040 for various other surgical procedures, enabling the user to perform coordinated treatment using two or more surgical instruments located at different points in a body cavity, so that each instrument can complete a part of the coordinated treatment even when the instruments cannot be directly visualized from each other as a result of being obscured from each other by surrounding tissue.

[0176] Figure 26 illustrates an exemplary process for aligning the first implant component 1080a with the second implant component 1080b using the EM-tracked distal ends of endoscopes 1020 and 1040, and multiple EM trackers (e.g., three or other numbers) on each of the first implant component 1080a and the second implant component 1080b. The “up,” neutral position, or orientation of the first endoscope 1020 is determined using the EM tracker on it. The determined orientation is indicated by arrow 1022 in Figure 26. The current orientation of the first implant component 1080a can be determined based on the location of the multiple EM trackers on it relative to the first endoscope 1020. The current orientation of the first implant component 1080a is indicated by arrow 1024 relative to arrow 1022. The current orientation of the second implant component 1080b can be determined using a similar process relating to the second endoscope 1040, and the first implant component 1080a and / or the second implant component 1080a can be realigned or re-oriented as necessary to align the two implant components 1080a, 1080b, as indicated by the solid arrow relating to the first implant component 1080a in Figure 26, ensuring successful alignment and deployment despite the obscured view between the two endoscopes 1020, 1040. In other exemplary embodiments, alignment indicators similar to those shown in Figure 26 may be incorporated into or extended to various displays used by the system 1000, such as a laparoscopic view available to the user.

[0177] Furthermore, while aligning the first implant component 1080a and the second implant component 1080b, the orientation of the instruments 1010, 1030 and the endoscopes 1020, 1040, as well as the characteristics of the tissue between them, are monitored by the controller 1050 to ensure that there is no accidental damage to the surrounding tissue, such as tissue twisting or blood flow obstruction, during deployment. For example, Figures 27, 28, and 29 illustrate exemplary deployment of implant components 1080a and 1080b. The controller 1050 displays a notification on the display 1060 and / or another display to inform the user that deployment of the two-part magnetic anastomosis device in Figure 27 should not yet be performed due to misalignment of orientation between the surrounding tissues on each side of the surgical treatment sites 1070a and 1070b, based in part on the identification of specific anatomical structures 1072a and 1072b located on each side of the surgical treatment sites 1070a and 1070b. In other embodiments, the controller 1050 can restrict the operation of instruments 1010 and 1030 to prevent deployment of the two-part magnetic anastomosis device due to orientation mismatch. In Figure 28, instruments 1010 and 1030 and endoscopes 1020 and 1040 assist in rotating the surrounding tissue to ensure proper anatomical orientation between surgical treatment sites 1070a and 1070b, and in Figure 29, the controller 1050 displays a notification on display 1060 and / or another display to inform the user that the surrounding tissue has been correctly oriented to each side of the surgical treatment sites 1070a and 1070b and that deployment of the two-part magnetic anastomosis device can proceed, thus ensuring proper blood flow and preventing potential twisting of the patient's intestines. In another embodiment, the controller 1050 can initially restrict the operation of instruments 1010 and 1030 to prevent deployment of the two-part magnetic anastomosis device due to misalignment of orientation. In this embodiment, the controller 1050 can re-enable the operation of instruments 1010 and 1030 to allow deployment of the two-part magnetic anastomosis device for correct orientation.In other embodiments, various other tissue manipulation approaches can be used to assist rotation, such as using another surgical instrument introduced into one or both of the surgical treatment sites 1070a, 1070b to provide rotational laparoscopy assistance, or assisting the rotation of the patient's intestine through the use of an intraluminal balloon.

[0178] The first implant component 1080a and the second implant component 1080b are deployed simultaneously when their orientation is determined again, they are rotated and aligned with each other based on any desired movement from the endoscopes 1020, 1040 and / or instruments 1010, 1030. Since the first implant component 1080a and the second implant component 1080b are magnetic, the first part 1080a and the second part 1080b connect with each other through the jejunal wall when deployed, forming the first treatment site 1070a and the second treatment site 1070b into a single connected or joined surgical treatment site.

[0179] Additional surgical instruments and / or additional scopes can be used in some surgical procedures to assist with additional visualization, movement of surrounding tissues, and implant positioning. The use of additional surgical instruments and / or additional scopes can be beneficial in certain embodiments because, given the length and complexity of operations through the intestines, it can be difficult to reach some target sites within the patient, such as target sites within the small intestine. For example, system 1000 can incorporate one or more additional surgical instruments and / or one or more additional scopes introduced into the patient's body from one or more additional access points, and the controller 1050 can coordinate the movement of the incorporated surgical instruments and / or scopes. Figure 23 illustrates, for example, a laparoscope 1085 with a field of view 1085a, indicated by a dashed line, introduced into the patient from a laparoscopic approach, and a gripper 1087 also introduced into the patient from a laparoscopic approach. In the illustrated embodiment, the laparoscope 1085 and the gripper 1087 can be used to determine and align the orientation of one or more of the instruments 1010, 1030, the endoscopes 1020, 1040, and the implant components 1080a, 1080b, in order to help ensure the successful deployment of the implant components 1080a, 1080b. For example, the gripper 1087 can be introduced to manipulate a portion of the small intestine so as to be visually indicated within the field of view 1085a of the laparoscope 1085, ensuring that each implant component 1080a, 1080b can reach the desired deployment location. Other instruments, such as dissection instruments, may be used to manipulate the portion of the small intestine.

[0180] In further embodiments, instead of or in addition to one or more surgical instruments, various surgical instruments and / or scopes, such as double-balloon enteroscopes and self-propelled dual-flex scopes, can be used to assist in navigating and controlling surrounding tissues, such as the small intestine, taking length and complexity into account. For example, if a user wants to position one of the endoscopes 1020, 1040 deeper into the patient's jejunum, a double-balloon enteroscope may be used, bundling a certain amount of small intestine onto the selected endoscope 1020, 1040 while preventing the selected endoscope 1020, 1040 from sliding or retracting backward from the jejunum. This approach allows several feet or more of small intestine to be bundled onto a selected endoscope 1020, 1040 that is several inches long, which is useful because the endoscope has a set length and the target surgical treatment site within the patient's small intestine may be further into the intestine than the length of the endoscope. In such embodiments, the user may need to use coordinated laparoscopy and endoscopic interactions to allow sufficient intestinal tissue to be pulled onto the selected endoscope 1020, 1040.

[0181] In other embodiments, a double-balloon enteroscope may be used to assist in deploying one of the implant components 1080a, 1080b of a two-part magnetic anastomosis device. For example, the most distal balloon of the enteroscope can be positioned distal to the second implant component 1080b to be deployed, and the balloon will increase visibility at the site when the most distal balloon is positioned at the second surgical treatment site 1070b. Additional details regarding various embodiments of the tissue manipulation instrument are provided, for example, in U.S. Patent Application No. 63 / 249,980 entitled “Cooperative Access,” filed on September 29, 2021. In some embodiments, various additional navigation aids may also be used, such as introducing a laparoscopic probe to a target site and / or otherwise tagging various sites within the patient virtually within a robotic system or through known means such as using reference markers. In various other embodiments, one or both of the implant components 1080a, 1080b of the two-part magnetic anastomosis device can be deployed using a laparoscopic approach instead of passing through one or both of the natural openings discussed above. Using a laparoscopic approach, one or both of the implant components 1080a, 1080b are deployed through the patient's jejunum or small intestine wall using coordinated movements between instruments 1010, 1030 and endoscopes 1020, 1040, as illustrated in Figure 30. This approach can utilize various laparoscopic access ports commonly used for anatomical mobilization to introduce one or both of the implant components 1080a, 1080b, thus avoiding the potentially difficult insertion process through the patient's intestine, and the implant components 1080a, 1080b can use tracking and imaging mechanisms similar to those discussed above to ensure accurate alignment and orientation before placement. Further details regarding various embodiments of the laparoscope are provided, for example, in U.S. Provisional Patent Application No. 63 / 249,980, entitled “Cooperative Access,” filed on September 29, 2021, as mentioned above.

[0182] Further details regarding partial jejunal detours and corresponding implants can be found in U.S. Patent No. 8,636,751, issued on January 28, 2014, entitled "Methods and devices for the rerouting of chyme to induce intestinal brake," U.S. Patent No. 10,206,682, issued on February 19, 2019, entitled "Magnetic tissue compression device with backup mechanical latch," U.S. Patent No. 10,517,600, issued on December 31, 2019, entitled "Magnetic anastomosis devices with varying magnetic force at a distance," U.S. Patent No. 10,779,831, issued on September 22, 2020, entitled "Systems, devices, and methods for forming anastomoses," and "Methods for partial diversion of the intestinal These are provided in U.S. Patent No. 11,033,272, entitled "tract", U.S. Patent Application Publication No. 2017 / 0265866, entitled "Targeting systems for providing accurate placement of magnetic anastomosis devices", published on 21 September 2017, and International Publication No. 2012007052(A1), entitled "A device for an endoluminal cholecysto-enterostomy", published on 19 January 2012, all of which are incorporated herein by reference in their entirety.

[0183] Those skilled in the art will recognize further features and advantages of the present invention based on the embodiments described above. Therefore, the present invention is not limited to what has been specifically shown and described, except as provided by the appended claims. All publications and references cited herein are expressly incorporated herein in their entirety by reference for all purposes.

[0184] [Implementation Method] (1) A system, A first surgical instrument, which is inserted into a first part of a body cavity and configured to act on a first surgical treatment site located within the body cavity of a patient, A second surgical instrument, inserted into a second portion of the body cavity and configured to act on a second surgical treatment site located within the body cavity, wherein the second portion of the body cavity differs from the first portion of the body cavity, and the second surgical treatment site differs from the first treatment tissue site. A first flexible endoscope having a first image sensor and configured such that the second surgical instrument is positioned so as not to be within the field of view of the first image sensor, A second flexible endoscope having a second image sensor, the first surgical instrument being positioned so that it is not within the field of view of the second image sensor, A system comprising: a controller configured to receive images collected by the first image sensor and the second image sensor, determine the first location of the first surgical instrument and the second location of the second surgical instrument relative to each other, and trigger synchronized surgical actions between the first surgical instrument and the second surgical instrument in the first and second treatment tissue sites, respectively. (2) A first portion of a surgical implant, which is releasably attached to the first surgical instrument and configured to be delivered into the body cavity while releasably attached to the first surgical instrument, The present invention further comprises a second portion of the surgical implant, which is releasably attached to the second surgical instrument and configured to be delivered into the body cavity while releasably attached to the second surgical instrument, The system according to Embodiment 1, wherein the controller is configured to simultaneously activate the deployment of the first and second portions of the surgical implant. (3) The system according to Embodiment 2, wherein the body cavity includes the jejunum and the surgical implant includes an anastomotic device. (4) The first portion of the surgical implant includes a first electromagnetic tracker configured to provide data relating to the first portion of the implant to the controller, The second portion of the surgical implant includes a second electromagnetic tracker configured to provide data relating to the second portion of the implant to the controller, The system according to embodiment 2 or 3, wherein the simultaneous deployment of the first and second portions by the controller is based on the received data relating to the first and second portions of the implant. (5) The system according to Embodiment 1, further comprising a third surgical instrument configured to be introduced into a third portion of the body cavity, and the controller configured to assist in causing the synchronized surgical actions of the first and second surgical instruments.

[0185] (6) The system according to Embodiment 5, wherein the first surgical instrument is configured to be introduced into the patient through a first natural orifice of the patient, the second surgical instrument is configured to be introduced into the patient through a second different natural orifice of the patient, and the third surgical instrument is configured to be introduced into the patient from a laparoscopic approach. (7) The system according to Embodiment 1, wherein the synchronized surgical action between the first surgical instrument and the second surgical instrument includes simultaneous synchronized surgical action at the first and second treatment tissue sites. (8) The system according to Embodiment 1, wherein the controller is configured to trigger the synchronized action between the first surgical instrument and the second surgical instrument when tissue obstructs the field of view of the first endoscope and when tissue obstructs the field of view of the second endoscope. (9) A system, Data processor, The system includes a memory that stores instructions configured to cause the processor to perform an operation, and the operation is, The first image sensor of the first endoscope receives first image data in real time that characterizes a first portion of the patient's body cavity, The second image data characterizing the second portion of the body cavity is received in real time from the second image sensor of the second endoscope, Based on the first image data, determine the first location of a first surgical instrument configured to act on tissue at a first surgical treatment site within the first portion of the body cavity, and determine that the first surgical instrument is outside the field of view of the second endoscope. Based on the second image data, the second location of the second surgical instrument relative to the first location of the first surgical instrument is determined, wherein the second surgical instrument is configured to act on tissue at the second surgical treatment site, and the second surgical instrument is outside the field of view of the first endoscope. A system comprising causing synchronized surgical action between the first surgical instrument and the second surgical instrument in a first and second tissue treatment site, respectively. (10) The system according to Embodiment 9, wherein the synchronized surgical action includes simultaneously deploying a first portion of the surgical implant from the first surgical instrument and a second portion of the surgical implant from the second surgical instrument.

[0186] (11) The system according to Embodiment 10, wherein the body cavity includes the jejunum, and the surgical implant includes a two-part magnetic anastomosis device. (12) The system according to any one of embodiments 9 to 11, further comprising receiving in real time from a third image sensor of the third endoscope third image data characterizing a third portion of the patient's body cavity. (13) The system according to Embodiment 9, wherein the synchronized surgical action includes avoiding penetration of any tissue by the first surgical instrument and the second surgical instrument. (14) A method, The first image sensor of the first endoscope system receives first image data in real time that characterizes a first portion of the patient's body cavity, The second image data characterizing the second portion of the body cavity is received in real time from the second image sensor of the second endoscope system, The controller determines, based on the first image data, the first location of a first surgical instrument used to manipulate tissue at a first surgical treatment site located within the first portion of the patient's body cavity, wherein the first surgical instrument is outside the field of view of the second endoscope system. The controller determines, based on the second image data, the second location of the second surgical instrument relative to the first surgical instrument, that the second surgical instrument manipulates tissue at a second surgical treatment site located within the second portion of the body cavity, and that the second surgical instrument is outside the field of view of the first endoscope system. A method comprising using the controller to trigger synchronized surgical actions between the first surgical instrument and the second surgical instrument in a first and second treatment tissue site, respectively. (15) The method of Embodiment 14, further comprising deploying a first portion of a surgical implant that is releasably attached to the first surgical instrument and configured to be delivered into the body cavity while being releasably attached to the first surgical instrument, and deploying a second portion of the surgical implant that is releasably attached to the second surgical instrument and configured to be delivered into the body cavity while being releasably attached to the second surgical instrument.

[0187] (16) The method according to embodiment 14 or 15, wherein the body cavity includes the jejunum and the surgical implant includes a two-part magnetic anastomosis device. (17) The method of Embodiment 14, further comprising receiving third image data characterizing a third portion of the patient's body cavity in real time from a third image sensor of the third endoscope. (18) A computer program product comprising instructions, wherein the instructions cause the system to perform the method described in Embodiment 14 when the program is executed by the controller of the system described in Embodiment 9. (19) A computer-readable medium storing the computer program product described in Embodiment 18. (20) A data transport signal for transporting the computer program product described in Embodiment 18.

Claims

1. It is a system, A first surgical instrument, which is inserted into a first part of a body cavity and configured to act on a first surgical treatment site located within the body cavity of a patient, A second surgical instrument, which is inserted into a second portion of the body cavity and configured to act on a second surgical treatment site located within the body cavity, wherein the second portion of the body cavity is different from the first portion of the body cavity, and the second surgical treatment site is different from the first surgical treatment site, A first flexible endoscope having a first image sensor and configured such that the second surgical instrument is positioned so as not to be within the field of view of the first image sensor, A second flexible endoscope having a second image sensor, wherein the first surgical instrument is positioned so that it is not within the field of view of the second image sensor, Controller and A first portion of a surgical implant, which is releasably attached to the first surgical instrument and configured to be delivered into the body cavity while releasably attached to the first surgical instrument, wherein the first portion of the surgical implant includes a first electromagnetic tracker configured to provide data relating to the first portion of the surgical implant to the controller, A second portion of the surgical implant is configured to be releasably attached to the second surgical instrument and to be delivered into the body cavity while releasably attached to the second surgical instrument, the second portion of the surgical implant includes a second electromagnetic tracker configured to provide data relating to the second portion of the surgical implant to the controller, The controller receives images collected by the first image sensor and the second image sensor, and when the second surgical instrument is inside the body cavity and not within the field of view of the first image sensor, and the first surgical instrument is inside the body cavity and not within the field of view of the second image sensor, it determines the location of the first electromagnetic tracker of the first surgical instrument and the location of the second electromagnetic tracker of the second surgical instrument relative to each other, and is configured to trigger synchronized surgical actions between the first surgical instrument and the second surgical instrument at the first surgical treatment site and the second surgical treatment site, respectively. The synchronized surgical action includes moving the first surgical instrument and the second surgical instrument relative to each other so that the first part is aligned with the second part.

2. The system according to claim 1, wherein the controller is configured to simultaneously activate the deployment of the first and second portions of the surgical implant.

3. The system according to claim 2, wherein the body cavity includes the jejunum, and the surgical implant includes an anastomotic device.

4. The system according to claim 2 or 3, wherein the simultaneous deployment of the first portion and the second portion by the controller is based on the received data relating to the first portion and the second portion of the surgical implant.

5. The system according to claim 1, further comprising a third surgical instrument configured to be introduced into a third portion of the body cavity, wherein the controller is configured to assist in causing the synchronized surgical actions of the first and second surgical instruments.

6. The system according to claim 5, wherein the first surgical instrument is configured to be introduced into the patient through a first natural orifice of the patient, the second surgical instrument is configured to be introduced into the patient through a second different natural orifice of the patient, and the third surgical instrument is configured to be introduced into the patient from a laparoscopic approach.

7. The system according to claim 1, wherein the synchronized surgical action between the first surgical instrument and the second surgical instrument includes simultaneous synchronized surgical action at the first surgical site and the second surgical site.

8. The system according to claim 1, wherein the controller is configured to trigger the synchronized action between the first surgical instrument and the second surgical instrument when tissue obstructs the field of view of the first flexible endoscope and when tissue obstructs the field of view of the second flexible endoscope.

9. The system according to claim 1, wherein the controller is configured to determine the orientation of the first part based on the location of the first electromagnetic tracker and to determine the orientation of the second part based on the location of the second electromagnetic tracker.

10. A third electromagnetic tracker disposed on the first flexible endoscope, A fourth electromagnetic tracker is positioned in the second flexible endoscope, The system according to claim 1, further comprising the following:

11. A fifth electromagnetic tracker arranged on the first surgical instrument, A sixth electromagnetic tracker is positioned on the second surgical instrument, The system according to claim 10, further comprising the following:

Images