Surgical system for giving and proving partial organ removals

Abstract

A surgical system for use in a surgical procedure is disclosed. The surgical system includes a surgical visualization system and a control circuit. The control circuit is configured to, based on visualization data from the surgical visualization system, identify a portion of an organ to be resected, determine a first value of a non-visualized parameter of the organ prior to resection of the portion, and determine a second value of the non-visualized parameter of the organ after resection of the portion. The resection of the portion is configured to produce an estimated volume reduction of the organ.

Description

Background Technology

[0001] Surgical systems are often integrated with imaging systems that allow clinicians to view the surgical site and / or one or more parts thereof on one or more monitors, such as a monitor. The monitors can be local to the operating room and / or remote. Imaging systems may include viewing mirrors with cameras that view the surgical site and transmit the view to a monitor accessible to the clinician. Viewing mirrors include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, colonoscopes, esophagogastric-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngoscopes-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and external endoscopes. Imaging systems may be limited by the information they can identify and / or convey to the clinician. For example, some imaging systems may not be able to identify certain hidden structures, physical contours, and / or dimensions in three-dimensional space during surgery. Additionally, some imaging systems may not be able to transmit and / or convey certain information to the clinician during surgery. Summary of the Invention

[0002] In one general aspect, a surgical system for surgical procedures is disclosed. The surgical system includes a surgical visualization system and control circuitry configured to, based on visualization data from the surgical visualization system, determine a portion of an organ to be removed, determine a first value of a non-visual parameter of the organ before removing the portion, and determine a second value of the non-visual parameter of the organ after removing the portion. The removal of the portion is configured to produce an estimated reduction in the organ's volume.

[0003] In another general aspect, a surgical system for surgical procedures is disclosed. The surgical system includes a surgical visualization system and control circuitry configured to receive input from a user indicating a portion of an organ to be removed, and to estimate the amount of organ volume reduction due to the removal of that portion based on visualization data from the surgical visualization system.

[0004] In yet another general aspect, a surgical system for surgical procedures is disclosed. The surgical system includes a surgical visualization system and control circuitry configured to receive first visualization data from the surgical visualization system in a first state of an organ, determine a first value of a non-visualization parameter of the organ in the first state, receive second visualization data from the surgical visualization system in a second state of the organ, determine a second value of the non-visualization parameter of the organ in the second state, and detect tissue abnormalities based on the first visualization data, the second visualization data, the first value of the non-visualization parameter, and the second value of the non-visualization parameter. Attached Figure Description

[0005] The novel features of various aspects are specifically set forth in the appended claims. However, the described aspects relating to both the organization and the method of operation are best understood by referring to the following description in conjunction with the accompanying drawings, wherein:

[0006] Figure 1 This is a schematic diagram of a surgical visualization system including an imaging device and a surgical apparatus according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify key structures beneath the surface of tissue.

[0007] Figure 2 It is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0008] Figure 2A A control circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0009] Figure 2B A combinational logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0010] Figure 2C A timing logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0011] Figure 3 It is described in accordance with at least one aspect of this disclosure. Figure 1 Triangulation is performed between the surgical device, imaging device, and key structure to determine the depth d of the key structure below the tissue surface. a A schematic diagram.

[0012] Figure 4 This is a schematic diagram of a surgical visualization system configured to identify critical structures beneath the tissue surface according to at least one aspect of this disclosure, wherein the surgical visualization system includes methods for determining the depth d of the critical structures beneath the tissue surface. a A pulsed light source.

[0013] Figure 5 This is a schematic diagram of a surgical visualization system including an imaging device and a surgical apparatus according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify key structures beneath the surface of tissue.

[0014] Figure 6 This is a schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, wherein the surgical visualization system is configured to identify key structures embedded within tissue.

[0015] Figure 7A and Figure 7B It is based on at least one aspect of this disclosure. Figure 6 A view of the key structure captured by a 3D camera, in which Figure 7A It is a view from the left lens of a 3D camera, and Figure 7B This is a view from the right lens of a 3D camera.

[0016] Figure 8 It is based on at least one aspect of this disclosure Figure 6 A schematic diagram of a surgical visualization system, in which the camera-key structure distance d from the 3D camera to the key structure can be determined. w .

[0017] Figure 9 This is a schematic diagram of a surgical visualization system that utilizes two cameras to determine the orientation of embedded key structures, according to at least one aspect of this disclosure.

[0018] Figure 10A This is a schematic diagram of a surgical visualization system utilizing a camera according to at least one aspect of this disclosure, the camera moving axially between multiple known orientations to determine the orientation of an embedded key structure.

[0019] Figure 10B It is based on at least one aspect of this disclosure Figure 10A A schematic diagram of a surgical visualization system in which a camera moves axially and rotationally between multiple known orientations to determine the orientation of embedded key structures.

[0020] Figure 11 It is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0021] Figure 12 This is a schematic diagram of a structured light source for a surgical visualization system according to at least one aspect of this disclosure.

[0022] Figure 13A It is a graph of the absorption coefficients of various biological materials at different wavelengths according to at least one aspect of this disclosure.

[0023] Figure 13B It is a schematic diagram of visualizing anatomical structures through a spectral surgical visualization system according to at least one aspect of this disclosure.

[0024] Figures 13C to 13E Exemplary hyperspectral identification features for distinguishing anatomical structures from obscuring objects, according to at least one aspect of this disclosure, are described, wherein Figure 13C It is a graphical representation of the characteristics of the ureter and its covering. Figure 13D It is a graphical representation of arterial features and obstructions, and Figure 13E It is a graphical representation of neural features and obscuring objects.

[0025] Figure 14 This is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distances to critical anatomical structures according to at least one aspect of this disclosure, the time-of-flight measurement system including a transmitter and a receiver (sensor) positioned on a common device.

[0026] Figure 15 It is based on at least one aspect of this disclosure Figure 14 A schematic diagram of the transmitted wave, received wave, and the delay between the transmitted and received waves of a NIR time-of-flight measurement system.

[0027] Figure 16 A NIR time-of-flight measurement system configured to sense distances from different structures, according to at least one aspect of the present disclosure, is shown. The time-of-flight measurement system includes a transmitter (transmitter) and a receiver (sensor) on separate devices.

[0028] Figure 17 This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of this disclosure.

[0029] Figure 18 It is a surgical system for performing surgical procedures in an operating room, according to at least one aspect of this disclosure.

[0030] Figure 19 An interactive surgical system implemented by a computer according to at least one aspect of the present disclosure is shown.

[0031] Figure 20 A diagram of a situational awareness surgical system according to at least one aspect of this disclosure is shown.

[0032] Figure 21 A timeline depicting the situational awareness of a hub according to at least one aspect of this disclosure is shown.

[0033] Figure 22 It is a logic flowchart of a process according to at least one aspect of this disclosure, which depicts a control program or logic configuration for associating visualization data with instrument data.

[0034] Figure 23 This is a schematic diagram of a surgical instrument according to at least one aspect of this disclosure.

[0035] Figure 24 It is a graph depicting a composite dataset according to at least one aspect of the present disclosure, along with virtual gauges of closing force (“FTC”) and firing force (“FTF”).

[0036] Figure 25AA general view of the screen of a visualization system according to at least one aspect of the present disclosure is shown, which displays the real-time feed of an end effector in the surgical field of view of a surgical procedure.

[0037] Figure 25B An enhanced view of the screen of a visualization system according to at least one aspect of the present disclosure is shown, which displays the real-time feed of an end effector in the surgical field of view of a surgical procedure.

[0038] Figure 26 It is a logic flowchart of a process according to at least one aspect of the present disclosure, which depicts a control program or logic configuration that synchronizes the motion of a virtual representation of an end effector component with the actual motion of the end effector component.

[0039] Figure 27 Anatomical structures in the body wall and cavity below the body wall according to at least one aspect of this disclosure are shown, wherein a cannula passes through the body wall into the cavity, and the screen displays the distance of the cannula from the anatomical structure, the risks associated with presenting surgical instruments through the cannula, and the estimated operation time associated therewith.

[0040] Figure 28 A virtual three-dimensional (“3D”) structure of a stomach exposed to structured light from a structured light projector, according to at least one aspect of this disclosure, is shown.

[0041] Figure 29 It is a logical flowchart of a process according to at least one aspect of this disclosure, which depicts a control procedure or logical configuration for associating visualization data with instrument data, wherein boxes with dashed lines indicate alternative implementations of the process.

[0042] Figure 30 A virtual 3D construct of a stomach exposed to structured light from a structured light projector, according to at least one aspect of this disclosure, is shown.

[0043] Figure 31 It is a logical flowchart of a process according to at least one aspect of the present disclosure, which depicts a control procedure or logical configuration for giving a resection path for removing a portion of an anatomical organ, wherein boxes with dashed lines indicate alternative specific implementations of the process.

[0044] Figure 32A A real-time view of the surgical field of view on the screen of a visualization system at the start of a surgical procedure is shown, according to at least one aspect of the present disclosure.

[0045] Figure 32B It is based on at least one aspect of this disclosure Figure 32AA magnified view of a portion of the surgical field of view, outlining the given surgical resection path superimposed on the surgical field of view.

[0046] Figure 32C It shows at least one aspect of this disclosure, forty-three minutes after the start of the surgical procedure. Figure 32B A real-time view of the surgical field of view.

[0047] Figure 32D At least one aspect of this disclosure is shown. Figure 32C A magnified view of the surgical field of view, outlining the modifications to the given surgical resection path.

[0048] Figure 33 It is a logical flowchart of a process according to at least one aspect of the present disclosure, which depicts a control procedure or logical configuration for presenting parameters of surgical instruments to or near a given surgical resection path, wherein boxes with dashed lines indicate alternative specific implementations of the process.

[0049] Figure 34 A virtual 3D structure of the stomach of a patient who has undergone sleeve gastrectomy according to at least one aspect of this disclosure is shown.

[0050] Figure 35 It shows Figure 34 A complete virtual resection of the stomach.

[0051] Figures 36A to 36C The firing of a surgical suturing instrument according to at least one aspect of this disclosure is shown.

[0052] Figure 37 It is a logic flowchart of a process according to at least one aspect of the present disclosure, which depicts a control program or logic configuration for adjusting the firing rate of a surgical instrument.

[0053] Figure 38 It is a logical flowchart of a process according to at least one aspect of the present disclosure, which depicts a control procedure or logical configuration for a given staple cartridge arrangement along a given surgical resection path.

[0054] Figure 39 It is a logical flowchart of a process according to at least one aspect of the present disclosure, which depicts a control procedure or logical configuration for giving surgical resection of an organ portion.

[0055] Figure 40 It is a logic flowchart of a process according to at least one aspect of the present disclosure, which depicts a control procedure or logic configuration for estimating the amount of organ volume reduction due to the removal of selected portions of an organ.

[0056] Figure 41A A patient's lung exposed to structured light according to at least one aspect of this disclosure is shown, including a portion to be removed during surgery.

[0057] Figure 41B This illustrates, according to at least one aspect of this disclosure, the process after the portion is removed. Figure 41A The patient's lungs.

[0058] Figure 41C The following figures, taken before and after the removal of that portion of the lung, are shown according to at least one aspect of this disclosure. Figure 41A and Figure 41B A curve showing the peak lung volume of the patient's lungs.

[0059] Figure 42 The diagram shows graphs of partial pressure of carbon dioxide (“PCO2”) in a patient’s lungs measured before, immediately after, and one minute after the removal of a portion of the lung, according to at least one aspect of this disclosure.

[0060] Figure 43 It is a logical flowchart of a process according to at least one aspect of this disclosure, which depicts a control procedure or logical configuration for detecting organizational anomalies using both visual and non-visual data.

[0061] Figure 44A The image shows a right lung in a first state according to at least one aspect of the present disclosure, wherein an imaging device emits a pattern of light onto its surface.

[0062] Figure 44B This illustrates a second state according to at least one aspect of this disclosure. Figure 44A The right lung, in which an imaging device projects a pattern of light onto its surface.

[0063] Figure 44C At least one aspect of this disclosure is shown. Figure 44A The top part of the right lung.

[0064] Figure 44D At least one aspect of this disclosure is shown. Figure 44B The top part of the right lung. Detailed Implementation

[0065] The applicant of this application holds the following U.S. patent applications, each of which is incorporated herein by reference in its entirety:

[0066] • The agent's case file number is END9228USNP1 / 190580-1M, and its title is "METHOD OF USING IMAGINGDEVICES IN SURGERY";

[0067] • The agent's case file number is END9227USNP1 / 190579-1, and its name is "ADAPTIVE VISUALIZATIONBY A SURGICAL SYSTEM";

[0068] • The agent's case file number is END9226USNP1 / 190578-1, and its title is "SURGICAL SYSTEM CONTROLBASED ON MULTIPLE SENSED PARAMETERS";

[0069] • The agent's case file number is END9225USNP1 / 190577-1, and its title is "ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS";

[0070] • The agent's case file number is END9224USNP1 / 190576-1, and its title is "ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS";

[0071] • The agent's case file number is END9223USNP1 / 190575-1, and its title is "SURGICAL SYSTEMSCORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA";

[0072] • The agent's case file number is END9222USNP1 / 190574-1, and its title is "SURGICAL SYSTEMS FORGENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLINGIDENTIFIED";

[0073] • The agent's case file number is END9221USNP1 / 190573-1, and its title is "SURGICAL SYSTEM FOROVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN";

[0074] • The agent's case file number is END9219USNP1 / 190571-1, and its title is "SYSTEM AND METHOD FORDETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE";

[0075] • The agent's case file number is END9218USNP1 / 190570-1, and its name is "VISUALIZATION SYSTEMSUSING STRUCTURED LIGHT";

[0076] • The agent's case file number is END9217USNP1 / 190569-1, and its name is "DYNAMIC SURGICALVISUALIZATION SYSTEMS"; and

[0077] • The agent's case file number is END9216USNP1 / 190568-1, and its title is "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM".

[0078] The applicant of this application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety:

[0079] • U.S. Patent Application Serial No. 16 / 354,417, entitled “INPUT CONTROLS FOR ROBOTICSURGERY”;

[0080] • U.S. Patent Application Serial No. 16 / 354,420, entitled “DUAL MODE CONTROLS FORROBOTIC SURGERY”;

[0081] • U.S. Patent Application Serial No. 16 / 354,422, entitled “MOTION CAPTURE CONTROLS FORROBOTIC SURGERY”;

[0082] • U.S. Patent Application Serial No. 16 / 354,440, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY”;

[0083] • U.S. Patent Application Serial No. 16 / 354,444, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TO TISSUE”;

[0084] • U.S. Patent Application Serial No. 16 / 354,454, entitled “ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS”;

[0085] • U.S. Patent Application Serial No. 16 / 354,461, entitled “SELECTABLE VARIABLE RESPONSEOF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS”;

[0086] • U.S. Patent Application Serial No. 16 / 354,470, entitled “SEGMENTED CONTROL INPUTS FORSURGICAL ROBOTIC SYSTEMS”;

[0087] • U.S. Patent Application Serial No. 16 / 354,474, entitled “ROBOTIC SURGICAL CONTROLSHAVING FEEDBACK CAPABILITIES”;

[0088] • U.S. Patent Application Serial No. 16 / 354,478, entitled “ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK”; and

[0089] • U.S. Patent Application Serial No. 16 / 354,481, entitled “JAW COORDINATION OF ROBOTICSURGICAL CONTROLS”.

[0090] The applicant of this application also owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety:

[0091] • U.S. Patent Application Serial No. 16 / 128,179, entitled “SURGICAL VISUALIZATION PLATFORM”;

[0092] • U.S. Patent Application Serial No. 16 / 128,180, entitled “CONTROLLING AN EMITTERASSEMBLY PULSE SEQUENCE”;

[0093] • U.S. Patent Application Serial No. 16 / 128,198, entitled “SINGULAR EMR SOURCE EMITTERASSEMBLY”;

[0094] • U.S. Patent Application Serial No. 16 / 128,207, entitled “COMBINATION EMITTER AND CAMERA ASSEMBLY”;

[0095] • U.S. Patent Application Serial No. 16 / 128,176, entitled “SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES”;

[0096] • U.S. Patent Application Serial No. 16 / 128,187, entitled “SURGICAL VISUALIZATION OF MULTIPLE TARGETS”;

[0097] • U.S. Patent Application Serial No. 16 / 128,192, entitled “VISUALIZATION OF SURGICALDEVICES”;

[0098] • U.S. Patent Application Serial No. 16 / 128,163, entitled “OPERATIVE COMMUNICATION OF LIGHT”;

[0099] • U.S. Patent Application Serial No. 16 / 128,197, entitled “ROBOTIC LIGHT PROJECTIONTOOLS”;

[0100] • U.S. Patent Application Serial No. 16 / 128,164, entitled “SURGICAL VISUALIZATIONFEEDBACK SYSTEM”;

[0101] • U.S. Patent Application Serial No. 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MONITORING”;

[0102] • U.S. Patent Application Serial No. 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA”;

[0103] • U.S. Patent Application Serial No. 16 / 128,170, entitled “ROBOTICALLY-ASSISTED SURGICALSUTURING SYSTEMS”;

[0104] • U.S. Patent Application Serial No. 16 / 128,183, entitled “SAFETY LOGIC FOR SURGICALSUTURING SYSTEMS”;

[0105] • U.S. Patent Application Serial No. 16 / 128,172, entitled “ROBOTIC SYSTEM WITH SEPARATEPHOTOACOUSTIC RECEIVER”; and

[0106] • US Patent Application Serial No. 16 / 128,185, entitled “FORCE SENSOR THROUGHSTRUCTURED LIGHT DEFLECTION”.

[0107] The applicant of this application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0108] • U.S. Patent Application Serial No. 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201111;

[0109] • U.S. Patent Application Serial No. 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTSFOR ROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201142;

[0110] • U.S. Patent Application Serial No. 15 / 940,711, entitled “SENSING ARRANGEMENTS FORROBOT-ASSISTED SURGICAL PLATFORMS”, now U.S. Patent Application Publication No. 2019 / 0201120; and

[0111] • U.S. Patent Application Serial No. 15 / 940,722, entitled “Characterization of Tissueir Reggaerities Through the Use of Mono-Chromatic Light Refraction”, now U.S. Patent Application Publication No. 2019 / 0200905.

[0112] The applicant of this patent application owns the following U.S. patent applications filed on December 4, 2018, the disclosure of each of which is incorporated herein by reference in its entirety:

[0113] • U.S. Patent Application Serial No. 16 / 209,395, entitled “METHOD OF HUB COMMUNICATION”, now U.S. Patent Application Publication No. 2019 / 0201136;

[0114] • U.S. Patent Application Serial No. 16 / 209,403, entitled “METHOD OF CLOUD BASED DATAANALYTICS FOR USE WITH THE HUB”, now U.S. Patent Application Publication No. 2019 / 0206569;

[0115] • U.S. Patent Application Serial No. 16 / 209,407, entitled “METHOD OF ROBOTIC HUBCOMMUNICATION, DETECTION, AND CONTROL”, now U.S. Patent Application Publication No. 2019 / 0201137;

[0116] • U.S. Patent Application Serial No. 16 / 209,416, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS”, now U.S. Patent Application Publication No. 2019 / 0206562;

[0117] • U.S. Patent Application Serial No. 16 / 209,423, entitled “METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THETISSUE WITHIN THE JAWS”, now U.S. Patent Application Publication No. 2019 / 0200981;

[0118] • U.S. Patent Application Serial No. 16 / 209,427, entitled “METHOD OF USING REINFORCEDFLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIOFREQUENCY DEVICES”, now U.S. Patent Application Publication No. 2019 / 0208641;

[0119] • U.S. Patent Application Serial No. 16 / 209,433, entitled “METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THESENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB”, now U.S. Patent Application Publication No. 2019 / 0201594;

[0120] • U.S. Patent Application Serial No. 16 / 209,447, entitled “METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB”, now U.S. Patent Application Publication No. 2019 / 0201045;

[0121] • U.S. Patent Application Serial No. 16 / 209,453, entitled “METHOD FOR CONTROLLING SMARTENERGY DEVICES”, now U.S. Patent Application Publication No. 2019 / 0201046;

[0122] • U.S. Patent Application Serial No. 16 / 209,458, entitled “METHOD FOR SMART ENERGYDEVICE INFRASTRUCTURE”, now U.S. Patent Application Publication No. 2019 / 0201047;

[0123] • U.S. Patent Application Serial No. 16 / 209,465, entitled “METHOD FOR ADAPTIVE CONTROLSCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION”, now U.S. Patent Application Publication No. 2019 / 0206563;

[0124] • U.S. Patent Application Serial No. 16 / 209,478, entitled “METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE”, now U.S. Patent Application Publication No. 2019 / 0104919;

[0125] • U.S. Patent Application Serial No. 16 / 209,490, entitled “METHOD FOR FACILITY DATACOLLECTION AND INTERPRETATION”, now U.S. Patent Application Publication No. 2019 / 0206564; and

[0126] • U.S. Patent Application Serial No. 16 / 209,491, entitled “METHOD FOR CIRCULAR STAPLERCONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS”, now U.S. Patent Application Publication No. 2019 / 0200998.

[0127] Before detailing the various aspects of the surgical visualization platform, it should be noted that the illustrative examples are not limited in application or use to the details of the construction and arrangement of the components shown in the accompanying drawings and specifications. The illustrative examples may be implemented or incorporated in other aspects, variations, and modifications, and may be practiced or performed in various ways. Furthermore, unless otherwise specified, the terminology and expressions used herein are chosen for the convenience of the reader in describing the illustrative examples and are not intended to be restrictive. Moreover, it should be understood that one or more of the aspects, expressions, and / or examples described below may be combined with any one or more of the other aspects, expressions, and / or examples described below.

[0128] Surgical visualization system

[0129] This disclosure relates to a surgical visualization platform that utilizes "digital surgery" to obtain additional information about a patient's anatomy and / or surgical procedures. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a helpful manner. For example, various aspects of this disclosure provide improved visualization of a patient's anatomy and / or surgical procedures.

[0130] "Digital surgery" can encompass robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and much more. While the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, they are not limited to such use. In some cases, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0131] In some cases, surgical systems incorporating surgical visualization platforms can enable intelligent anatomy to identify and avoid critical structures. Critical structures include anatomical structures such as the ureter, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other cases, critical structures can be, for example, foreign structures within the anatomical field, such as surgical devices, surgical fasteners, clamps, pins, probes, bands, and / or plates. Critical structures can be determined based on different patients and / or different surgical procedures. Exemplary critical structures are also described herein. For example, intelligent anatomy techniques can provide intraoperative guidance for improved anatomy and / or enable intelligent decision-making using critical anatomical structure detection and avoidance techniques.

[0132] Surgical systems incorporating surgical visualization platforms can also enable intelligent anastomosis techniques, which utilize improved workflows to provide more consistent anastomosis at optimal locations. The various surgical visualization platforms and procedures described herein can also be used to improve cancer localization techniques. For example, cancer localization techniques can identify and track the location, orientation, and boundaries of cancer. In some cases, cancer localization techniques can compensate for movement of instruments, the patient, and / or the patient's anatomy during surgery to provide clinicians with guidance back to the point of interest.

[0133] In certain aspects of this disclosure, surgical visualization platforms can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly when dissecting tissues and / or placing suture devices. Some of the tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. Regarding lymph node diagnosis and mapping, surgical visualization platforms can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0134] During surgery, information available to clinicians via the naked eye and / or imaging systems can provide an incomplete view of the surgical site. For example, certain structures (such as those embedded or buried within organs) may be at least partially concealed or hidden from view. Additionally, certain dimensions and / or relative distances may be difficult to detect using existing sensor systems and / or difficult to perceive with the naked eye. Furthermore, some structures may be mobile preoperatively (e.g., before surgery but after preoperative scanning) and / or intraoperatively. In such cases, clinicians may be unable to accurately pinpoint the location of critical structures intraoperatively.

[0135] Clinicians' decision-making processes can be hampered when the location of critical structures is uncertain and / or when the proximity of critical structures to surgical instruments is unknown. For example, clinicians may avoid certain areas to prevent accidental dissection of critical structures; however, the avoided areas may be unnecessarily large and / or at least partially misaligned. Due to uncertainty and / or excessive caution, clinicians may be unable to access certain desired areas. For instance, excessive caution may lead clinicians to leave portions of tumor and / or other unwanted tissue in an attempt to avoid critical structures, even if the critical structures are not in that particular area and / or will not be negatively affected by a clinician working in that particular area. In some cases, surgical outcomes can be improved by increasing knowledge and / or certainty, which can enable surgeons to be more accurate in specific anatomical areas and, in other cases, make surgeons less conservative / more aggressive.

[0136] In various aspects, this disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, this disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what clinicians can see with the "naked eye" and / or beyond what imaging systems can identify and / or convey to clinicians. Various surgical visualization systems can enhance and strengthen what clinicians can know before tissue treatment (e.g., dissection) and thus improve outcomes in a variety of situations.

[0137] For example, a visualization system may include a first light emitter configured to emit multiple spectral waves, a second light emitter configured to emit light patterns, and one or more receivers or sensors configured to detect visible light, molecular responses to spectral waves (spectral imaging), and / or light patterns. It should be noted that throughout the entire disclosure below, unless specifically mentioned as visible light, any reference to “light” may include photons in the visible and / or invisible portions of electromagnetic radiation (EMR) or the EMR wavelength spectrum. Surgical visualization systems may also include an imaging system and control circuitry that communicates signals with the receiver and the imaging system. Based on the output from the receiver, the control circuitry may determine a geometric surface mapping (i.e., three-dimensional surface topography) of the visible surface at the surgical site and one or more distances relative to the surgical site. In some cases, the control circuitry may determine one or more distances to at least partially concealed structures. Furthermore, the imaging system may communicate the geometric surface mapping and one or more distances to the clinician. In such cases, the enhanced view of the surgical site provided to the clinician can represent concealed structures within the relevant environment of the surgical site. For example, an imaging system can virtually enhance occult structures on a geometric surface mapping of occult and / or obstructing tissue, similar to lines drawn on the ground to indicate practical lines beneath the surface. Additionally or alternatively, the imaging system can convey the proximity of one or more surgical instruments to visible obstructing tissue and / or to at least partially occulted structures, and / or the depth of the occult structure beneath the visible surface of the obstructing tissue. For example, a visualization system can determine the distance to an enhancing line relative to the surface of visible tissue and convey that distance to the imaging system.

[0138] In various aspects of this disclosure, surgical visualization systems for intraoperative identification and avoidance of critical structures are disclosed. Such surgical visualization systems can provide valuable information to clinicians during surgical procedures. Thus, for example, the clinician knows that the surgical visualization system is tracking, for example, critical structures accessible during anatomy (such as the ureter, specific nerves, and / or critical blood vessels), and can confidently maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide the clinician with instructions for a sufficiently long period to allow the clinician to pause and / or slow down the surgery and assess proximity to critical structures to prevent accidental damage. The surgical visualization system can provide the clinician with an ideal, optimized, and / or customizable amount of information to allow the clinician to confidently and / or rapidly maneuver through tissues while avoiding accidental damage to healthy tissues and / or critical structures, and thus minimizing the risk of injury caused by the surgery.

[0139] Figure 1This is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In some cases, the surgical visualization system 100 can be used intraoperatively to provide clinicians with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during surgical procedures. The surgical visualization system 100 is configured to identify critical structures intraoperatively and / or facilitate the avoidance of critical structures 101 by surgical instruments. For example, by identifying critical structures 101, clinicians can avoid manipulating surgical instruments around areas within a predetermined proximity of critical structures 101 during surgery. For example, clinicians can avoid dissecting veins, arteries, nerves, and / or blood vessels, such as those identified as critical structures 101, and / or avoid dissecting near these critical structures. In various cases, the critical structure 101 can be determined based on different patients and / or different surgical procedures.

[0140] The surgical visualization system 100 incorporates a distance sensor system 104 that integrates tissue identification and geometric surface mapping. Combined, these features of the surgical visualization system 100 can determine the orientation of a key structure 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the key structure 101. Furthermore, the surgical visualization system 100 includes an imaging system comprising, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various cases, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Views from the imaging device 120 can be provided to a clinician, and in various aspects of this disclosure, these views can be enhanced with additional information based on tissue identification, topographic mapping, and the distance sensor system 104. In such cases, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide clinicians with advanced data synthesis and integration information during surgery.

[0141] Imaging devices may include cameras or imaging sensors configured to detect, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). In various aspects of this disclosure, imaging systems may include, for example, imaging devices such as endoscopes. Additionally or alternatively, imaging systems may include, for example, imaging devices such as arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, colonoscopes, esophagogastric-duodenoscopes (gastroscopes), laryngoscopes, nasopharyngoscopes-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, or external endoscopes. In other cases, such as in open surgical applications, the imaging system may not include an observation endoscope.

[0142] In all respects of this disclosure, the tissue identification subsystem can be implemented using a spectral imaging system. The spectral imaging system may rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” published March 1, 2016, the entire contents of which are incorporated herein by reference.

[0143] In various aspects of this disclosure, the surface mapping subsystem can be implemented using an optical patterning system, as further described herein. The use of optical patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be used in the surgical visualization system described herein.

[0144] Structured light is the process of projecting a known pattern (typically a grid or horizontal stripes) onto a surface. U.S. Patent Application Publication 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published March 2, 2017, and U.S. Patent Application Publication 2017 / 0251900, entitled "DEPICTION SYSTEM," published September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. The full text of U.S. Patent Application Publication 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," and U.S. Patent Application Publication 2017 / 0251900, entitled "DEPICTION SYSTEM," published September 7, 2017, is incorporated herein by reference.

[0145] In various aspects of this disclosure, the distance determination system can be incorporated into a surface mapping system. For example, structured light can be used to generate a three-dimensional virtual model of a visible surface and determine various distances relative to the visible surface. Alternatively or additionally, the distance determination system can rely on time-of-flight measurements to determine one or more distances to tissue (or other structures) identified at a surgical site.

[0146] Figure 2 This is a schematic diagram of a control system 133 that can be used with a surgical visualization system 100. The control system 133 includes control circuitry 132 that communicates signalally with a memory 134. The memory 134 stores instructions executable by the control circuitry 132 to determine and / or identify critical structures (e.g., Figure 1 The key structure 101 in the system determines and / or calculates one or more distances and / or three-dimensional digital representations, and transmits certain information to one or more clinicians. For example, memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. Control system 133 also includes imaging system 142 having one or more cameras 144 (e.g., ...). Figure 1 The camera 144 may include one or more imaging devices 120, one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 may include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, three-dimensional lenses, etc.) that emit light in the various visible and invisible spectra. The display 146 may include one or more screens or monitors for displaying real, virtual, and / or virtual-enhanced images and / or information to one or more clinicians.

[0147] At the heart of the camera 144 is the image sensor 135. Generally, a modern image sensor 135 is a solid-state electronic device containing up to millions of discrete photodetector sites (called pixels). Image sensor 135 technology falls into one of two categories: charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS) imagers, and recently, short-wave infrared (SWIR) has become an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) with bumps bonded to the CCD imaging substrate. Both CCD and CMOS image sensors 135 are sensitive to wavelengths from approximately 350 nm to 1050 nm, although this range is typically given as 400 nm to 1000 nm. Generally, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and therefore cannot distinguish colors. Therefore, two types of color CCD cameras exist: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to split incident light into a series of colors and employing interpolation algorithms to resolve the panchromatic image. Each color is then assigned to a different set of pixels. Three-chip color CCD cameras provide higher resolution by using prisms to direct each portion of the incident spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value, rather than using algorithms to determine color. Three-chip cameras offer extremely high resolution.

[0148] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source may be pulsed to emit wavelengths of light within the range of the spectral light source 150 and the range of light within the range of the structured light source 152. Alternatively, a single light source may be pulsed to provide wavelengths of light in the invisible spectrum (e.g., infrared light) and light in the visible spectrum. The spectral light source 150 may be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic unit 140 may identify critical structures via data received from the spectral light source 150 by the image sensor 135 portion of the camera 144. The surface mapping logic unit 136 may determine the surface profile of the visible tissue based on the reflected structured light. Using the time-of-flight measurement results, the distance determination logic unit 141 may determine one or more distances to the visible tissue and / or critical structure 101. One or more outputs from the surface mapping logic unit 136, the tissue identification logic unit 140, and the distance determination logic unit 141 may be provided to the imaging logic unit 138 and may be combined, mixed, and / or overlapped to be communicated to a clinician via the display 146 of the imaging system 142.

[0149] The instruction manual now briefly goes to Figures 2A to 2C This describes various aspects of the control circuitry 132 used to control various aspects of the surgical visualization system 100. (Go to...) Figure 2A This illustration shows a control circuit 400 configured to control various aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure. The control circuit 400 may be configured to implement the various processes described herein. The control circuit 400 may include a microcontroller including one or more processors 402 (e.g., microprocessor, microcontroller) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions to implement the various processes described herein. The processor 402 may be any of a variety of single-core or multi-core processors known in the art. The memory circuit 404 may include volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0150] Figure 2B A combinational logic circuit 410, configured to control various aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure, is shown. The combinational logic circuit 410 may be configured to implement the various processes described herein. The combinational logic circuit 410 may include a finite state machine including a combinational logic component 412 configured to receive data associated with a surgical instrument or tool at input 414, process the data through the combinational logic component 412, and provide an output 416.

[0151] Figure 2C A sequential logic circuit 420, configured to control various aspects of a surgical visualization system 100 according to at least one aspect of this disclosure, is shown. The sequential logic circuit 420 or combinational logic element 422 may be configured to implement the various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, combinational logic element 422, at least one memory circuit 424, and a clock 429. At least one memory circuit 424 may store the current state of the finite state machine. In some cases, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic element 422 is configured to receive data associated with a surgical device or system from input 426, process the data through the combinational logic element 422, and provide an output 428. In other aspects, the circuit may include a processor (e.g., Figure 2AThe various processes described herein are implemented by combining a processor 402 and a finite state machine. In other aspects, the finite state machine may include combinational logic circuits (e.g., combinational logic circuit 410, ...). Figure 2B The combination of ) and sequential logic circuit 420.

[0152] See you again Figure 1 In the surgical visualization system 100, the key structure 101 can be an anatomical structure of interest. For example, the key structure 101 can be an anatomical structure such as the ureter, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other cases, the key structure 101 can be, for example, an external structure in the anatomical field, such as a surgical device, surgical fastener, clamp, pin, probe, band, and / or plate. Exemplary key structures are further described herein and in co-filed U.S. patent applications (including, for example, U.S. Patent Application No. 16 / 128,192, filed September 11, 2018, entitled “VISUALIZATION OF SURGICAL DEVICES”), the entire contents of which are incorporated herein by reference.

[0153] In one aspect, the key structure 101 may be embedded within the tissue 103. In other words, the key structure 101 may be located below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the key structure 101 from view by the clinician. From the perspective of the imaging device 120, the key structure 101 is also obscured by the tissue 103. The tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other cases, the key structure 101 may be partially obscured, making it invisible.

[0154] Figure 1 Surgical device 102 is also depicted. Surgical device 102 includes an end effector having opposing jaws extending from the distal end of the axis of surgical device 102. Surgical device 102 can be any suitable surgical device, such as, for example, a dissecting instrument, suture device, gripper, applicator, and / or energy device (including monopolar probes, bipolar probes, ablation probes, and / or ultrasound end effectors). Alternatively or additionally, surgical device 102 may include, for example, another imaging or diagnostic modality, such as an ultrasound device. In one aspect of this disclosure, surgical visualization system 100 can be configured to enable the identification of one or more key structures 101 and the proximity of surgical device 102 to key structures 101.

[0155] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (such as a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens used together to simultaneously record two two-dimensional images, and thus generate a three-dimensional image of the surgical site, render the three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicating the morphology of visible tissue and the identification and orientation of hidden key structures, as further described herein. For example, the field of view of the imaging device 120 may overlap with a pattern of light (structured light) on the surface 105 of the tissue, such as… Figure 1 As shown.

[0156] In one aspect, the surgical visualization system 100 may be integrated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 include rigid structural members 116 and joints 118, which may include servo motor controls. The first robotic arm 112 is configured to manipulate a surgical device 102, and the second robotic arm 114 is configured to manipulate an imaging device 120. A robot control unit may be configured to issue control movements to the robotic arms 112 and 114, which may affect, for example, the surgical device 102 and the imaging device 120.

[0157] The surgical visualization system 100 also includes an emitter 106 configured to emit patterns of light, such as stripes, grid lines, and / or dots, to enable the determination of the topography or topography of surface 105. For example, a projection light array 130 can be used for three-dimensional scanning and registration on surface 105. The projection light array 130 can be emitted from the emitter 106 located, for example, on one of surgical devices 102 and / or robotic arms 112, 114 and / or imaging device 120. In one aspect, the projection light array 130 is used to determine the shape defined by the surface 105 of tissue 103 and / or the movement of the surface 105 during surgery. The imaging device 120 is configured to detect the projection light array 130 reflected from surface 105 to determine the topography of surface 105 and various distances relative to surface 105.

[0158] In one aspect, the imaging device 120 may further include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positioned by a robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, spectrometer, or vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable the identification of the type of anatomical and / or physical structures (such as the critical structure 101). Identification of the critical structure 101 may be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform transmitter 123 may include, for example, a photoacoustic imaging system. In other cases, the optical waveform transmitter 123 may be positioned on a surgical device separate from the imaging device 120.

[0159] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at a surgical site. In one aspect, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including a transmitter (such as transmitter 106) and a receiver 108 positionable on the surgical device 102. In other cases, the time-of-flight transmitter may be separate from the structured light transmitter. In a general aspect, the transmitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a mating sensor. The time-of-flight distance sensor system 104 can detect the “time of flight,” or the time taken for the laser emitted by transmitter 106 to bounce back to the sensor portion of receiver 108. The use of a very narrow light source in transmitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of tissue 103 directly in front of the distance sensor system 104. See still. Figure 1 d e It is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and d t This refers to the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. The distance sensor system 104 can be used to determine the transmitter-tissue distance d. e Device-tissue distance d tThe device-tissue distance d can be obtained from the known orientation of the transmitter 106 relative to the distal end of the surgical device 102 on its axis. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d is... t Based on the transmitter-to-organization distance d e Determined. In some cases, the axis of the surgical device 102 may include one or more articulated joints and may be capable of articulation relative to the transmitter 106 and the jaws. The articulated configuration may include, for example, a multi-joint vertebral structure. In some cases, a three-dimensional camera may be used to triangulate one or more distances to surface 105.

[0160] In various cases, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate surgical device rather than on surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which surgical device 102 extends to reach the surgical site. In other cases, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate robot-controlled arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In some cases, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance from the transmitter 106 on surgical device 102 to the surface 105 of tissue 103 using a line between the transmitter 106 on surgical device 102 and the imaging device 120. For example, the distance d may be determined based on the known orientation of the transmitter 106 (on surgical device 102) and the receiver 108 (on imaging device 120) of the time-of-flight distance sensor system 104. e Triangulation is performed. The three-dimensional orientation of receiver 108 can be known and / or registered with the robot coordinate plane during surgery.

[0161] In some cases, the orientation of the transmitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the orientation of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other cases, the surgical visualization system 100 can be used separately from the robotic system. In such cases, the distance sensor system 104 can operate independently of the robotic system.

[0162] In some cases, one or more of the robotic arms 112, 114 may be detached from the main robotic system used in the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and registered with a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or register the orientation of the robotic arms 112, 114 relative to a specific coordinate system. Similarly, the orientation of the surgical device 102 and the imaging device 120 may be registered with a specific coordinate system.

[0163] See still Figure 1 d w It is the camera-critical structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the critical structure 101, and d A This is the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time-of-flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d. w The use of spectral imaging combined with a time-of-flight sensor is further described in this paper. Additionally, see now... Figure 3 In various aspects of this disclosure, the depth d of the key structure 101 relative to the surface 105 of the tissue 103 A This can be determined by the following method: based on the distance d w and the known orientations of transmitter 106 on surgical device 102 and optical waveform transmitter 123 on imaging device 120 (and therefore the known distance d between them). x Triangulation is performed to determine the distance d. y (where d is the distance) e and d A sum).

[0164] Alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or waveform range) can be used to determine the camera-critical structure distance d. w Furthermore, the second waveform (or waveform range) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0165] Alternatively or alternatively, in some cases, distance d A It can be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance dA Spectral imaging can be used to determine this because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first manner or in a first amount, and collagen can reduce the detection signal in a different second manner or in a second amount.

[0166] See now Figure 4 The surgical visualization system 160 includes a surgical device 162 comprising an optical waveform transmitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform transmitter 123 may be configured to emit waveforms for determining a distance d from a common device (such as the surgical device 162). t and d w As further described herein. In such cases, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is... A This can be determined as follows:

[0167] d A =d w -d t .

[0168] As disclosed herein, various information regarding visible tissue, embedded key structures, and surgical devices can be determined using a combined approach that integrates an image sensor configured to detect spectral wavelengths and structured light arrays with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Furthermore, the image sensor can be configured to receive visible light and thus provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and to render a three-dimensional image of the surface tissue and underlying anatomical structures. In various cases, the imaging device 120 may include multiple image sensors.

[0169] Camera-critical structure distance d w Detection can also be performed using one or more alternative methods. In one aspect, key structures 201 can be illuminated using techniques such as fluorescence visualization (e.g., fluorescent indocyanine green (ICG)). Figures 6 to 8 As shown. Camera 220 may include two optical waveform sensors 222 and 224, which simultaneously capture left and right images of the key structure 201. Figure 7A and Figure 7B In such cases, camera 220 can depict the glow of the key structure 201 beneath the surface 205 of tissue 203, and at a distance of d. wThe distance can be determined from the known distance between sensors 222 and 224. In some cases, the distance can be determined more accurately by using more than one camera or by moving the camera between multiple positions. In some aspects, one camera may be controlled by a first robotic arm, and a second camera may be controlled by another robotic arm. In such robotic systems, a camera may be, for example, a follower camera on a follower arm. The follower arm and the camera on it may be programmed to track another camera and maintain, for example, a specific distance and / or lens angle.

[0170] In other aspects, the surgical visualization system 100 may employ two separate waveform receivers (i.e., camera / image sensor) to determine d w See now. Figure 9 If the critical structure 301 or its contents (e.g., blood vessels or vascular contents) can emit signals 302 using fluorescence fluoroscopy, the actual location can be triangulated based on two separate cameras 320a, 320b at a known location.

[0171] On the other hand, see now Figure 10A and Figure 10B The surgical visualization system can use a shake or move camera 440 to determine the distance d. w Camera 440 is robotically controlled, allowing its three-dimensional coordinates at different orientations to be known. In various situations, camera 440 can pivot at the cannula or patient interface. For example, if a critical structure 401 or its contents (e.g., a blood vessel or its contents) can emit signals, such as using fluorescence fluoroscopy, the actual position can be triangulated based on camera 440 rapidly moving between two or more known locations. Figure 10A In this process, camera 440 moves axially along axis A. More specifically, camera 440 translates a distance d1 along axis A closer to critical structure 401 to a position indicated as location 440', such as by moving in and out on a robotic arm. The distance to critical structure 401 can be calculated as camera 440 moves a distance d1 and the size of the view changes relative to critical structure 401. For example, an axial translation of 4.28 mm (distance d1) could correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Alternatively or additionally, camera 440 can rotate or sweep along arcs between different orientations. See now. Figure 10B Camera 440 moves axially along axis A and rotates about axis A by an angle θ3. The pivot point 442 for the rotation of camera 440 is located at the cannula / patient interface. Figure 10B In this process, camera 440 is translated and rotated to position 440". As camera 440 is moved and the view edges change with respect to key structure 401, the distance to key structure 401 can be calculated. Figure 10BIn this case, the distance d2 can be, for example, 9.01 mm, and the angle θ3 can be, for example, 0.9 degrees.

[0172] Figure 5 A surgical visualization system 500 is depicted, which is similar to surgical visualization system 100 in many respects. In various cases, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light of multiple wavelengths to obtain spectral images of, for example, hidden structures. In various cases, imaging device 520 may also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown intraoperatively for identifying and facilitating the avoidance of certain critical structures not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (uterus in this example).

[0173] The surgical visualization system 500 is configured to determine the emitter-tissue distance d from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to visualize based on the transmitter-tissue distance d. e The device extends from the surgical device 502 to the surface 505 of the uterus 503 via a tissue distance d. t The surgical visualization system 500 is also configured to determine the tissue-ureteral distance d from the ureter 501a to the surface 505. A And the camera-ureter distance d from imaging device 520 to ureter 501a w As this article discusses... Figure 1 For example, the surgical visualization system 500 may utilize, for example, spectral imaging and time-of-flight sensors to determine the distance d. w In various situations, the surgical visualization system 500 can determine (e.g., triangulation) the tissue-ureter distance d based on other distance and / or surface mapping logic components described herein. A (or depth).

[0174] See now Figure 11The diagram depicts a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For instance, the control system 600 is a conversion system that integrates spectral feature tissue recognition and structured optical tissue localization to identify key structures, particularly when these structures are obscured by other tissues such as fat, connective tissue, blood, and / or other organs. Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue within an organ from healthy tissue.

[0175] The control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are utilized to detect and identify anatomical structures in a surgical field of view. The control system 600 includes conversion logic circuitry 648 to transform tissue data into information usable by the surgeon. For example, key structures within the anatomical structure can be identified using variable reflectivity based on the wavelength relative to the masking material. Furthermore, the control system 600 combines the identified spectral features and structured light data into an image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical use in a system with enhanced image overlay. Additional visual information can be used to employ the technology both intraoperatively and preoperatively. In various situations, the control system 600 is configured to provide warnings to clinicians upon approach to one or more key structures. Various algorithms can be employed to guide robotic automation and semi-automation methods based on surgical procedures and proximity to key structures.

[0176] Projected light arrays are used to determine tissue shape and movement intraoperatively. Alternatively, flash lidar can be used for surface mapping of tissue.

[0177] The control system 600 is configured to detect critical structures and provide image overlays of the critical structures, and to measure distances to the surface of visible tissue and to the embedded / buried critical structures. In other cases, the control system 600 may measure distances to the surface of visible tissue or detect critical structures and provide image overlays of the critical structures.

[0178] The control system 600 includes a spectral control circuit 602. For example, the spectral control circuit 602 may be a field-programmable gate array (FPGA) or, as described herein, a... Figures 2A to 2CAnother suitable circuit configuration is described. The spectral control circuit 602 includes a processor 604 to receive video input signals from a video input processor 606. For example, the processor 604 may be configured for hyperspectral processing and may utilize C / C++ code. For example, the video input processor 606 receives video input control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signals from the video input processor 606 and provide video output signals to a video output processor 608, which includes, for example, hyperspectral video output of interface control (metadata) data. The video output processor 608 provides the video output signals to an image overlay controller 610.

[0179] A video input processor 606 is coupled to a camera 612 on the patient side via a patient isolation circuit 614. As previously described, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers to isolate the patient from other circuitry in the system. The camera 612 receives intraoperative images via optics 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include, for example, [other types described herein]. Figure 2 Any image sensor technology described herein. In one aspect, camera 612 outputs an image with a 14-bit / pixel signal. It should be understood that higher or lower pixel resolutions may be used without departing from the scope of this disclosure. An isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which utilizes hardware register 618 and a Nios2 coprocessor 620 to process the camera output signal 613. The color RGB fused output signal is provided to a video input processor 606 and a laser pulse control circuit 622.

[0180] Laser pulse control circuit 622 controls laser engine 624. Laser engine 624 outputs light of multiple wavelengths (λ1, λ2, λ3...λn), including near-infrared (NIR). Laser engine 624 can operate in multiple modes. In one aspect, laser engine 624 can operate in, for example, two modes. In the first mode (e.g., normal operation mode), laser engine 624 outputs an illumination signal. In the second mode (e.g., identification mode), laser engine 624 outputs RGBG and NIR light. In various cases, laser engine 624 can operate in polarization mode.

[0181] Light output 626 from laser engine 624 illuminates the target anatomical structure in surgical site 627 during surgery. Laser pulse control circuitry 622 also controls laser pulse controller 628 for laser patterning projector 630, which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) onto the surgical tissue or organ at surgical site 627. Camera 612 receives the patterned light and reflected light output through camera optics 632. Image sensor 634 converts the received light into digital signals.

[0182] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and the video input module 636 for reading the laser pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser pattern 631 and outputs a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0183] The first video output signal 640 and the second video output signal 642 include data representing the orientation of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. Combined with data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine the distance d to the buried critical structure. A ( Figure 1 (For example, via triangulation algorithm 644), and distance d A The video output processor 646 can provide the output to the image overlay controller 610. The aforementioned conversion logic components may include the conversion logic circuit 648, the intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0184] In various situations, preoperative data 650 from CT or MRI scans can be used to register or match certain three-dimensional deformable tissues. This preoperative data 650 can be provided to an integration module 643 and ultimately to an image overlay controller 610, allowing this information to be overlaid with a view from a camera 612 and provided to a video monitor 652. The registration of preoperative data is further described herein and in the foregoing concurrently filed U.S. patent applications (including, for example, U.S. Patent Application No. 16 / 128,195, filed September 11, 2018, entitled “INTEGRATION OF IMAGING DATA”), the entire contents of which are incorporated herein by reference.

[0185] Video monitor 652 can output an integrated / enhanced view from image overlay controller 610. Clinicians can select and / or switch between different views on one or more monitors. On the first monitor 652a, a clinician can switch between (A) a view depicting a 3D rendering of visible tissue and (B) an enhanced view depicting one or more hidden critical structures on the 3D rendering of visible tissue. On the second monitor 652b, a clinician can, for example, switch distance measurements to the surface of one or more hidden critical structures and / or visible tissue.

[0186] The control system 600 and / or its various control circuits can be integrated into the various surgical visualization systems disclosed herein.

[0187] Figure 12 A structured (or patterned) light system 700 according to at least one aspect of this disclosure is illustrated. As described herein, structured light in the form of stripes or lines can be projected, for example, from a light source and / or a projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contour of the surface 705. In various aspects, it may be similar to imaging device 120 ( Figure 1 The camera 720 can be configured, for example, to detect the pattern of light projected onto the surface 705. The way the projected pattern deforms upon impact with the surface 705 allows the vision system to calculate depth and surface information of the target's anatomy.

[0188] In some cases, invisible (or imperceptible) structured light can be used, where it can be employed without interfering with other computer vision tasks where the projected pattern might become confuse. For example, alternating infrared light or extremely fast visible light frame rates between two completely opposite patterns can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0189] As described above, the various surgical visualization systems described herein can be used to visualize various types of tissues and / or anatomical structures, including those that are obscured by EMR in the visible portion of the spectrum and thus cannot be visualized. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissues based on different combinations of constituent materials. Specifically, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the absorption coefficients of the tissue at various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on specific combinations of constituent materials. For illustration, Figure 13AThis is graph 2300 depicting how the absorption coefficients of various biomaterials vary across the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient of the biomaterial (e.g., in cm⁻¹). -1 The horizontal axis 2304 represents the EMR wavelength (e.g., in μm). Graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of proteins at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxyhemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxyhemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials; therefore, tissue types visualized by a surgical visualization system can be identified and distinguished based on specific combinations of detected constituent materials. Thus, a spectral imaging system can be configured to emit multiple different wavelengths of EMR, determine the constituent materials of the tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on specific detection combinations of constituent materials.

[0190] Figure 13B This demonstrates the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures. Figure 13B In this imaging system, a spectral emitter 2320 (e.g., a spectral light source 150) is used to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissue and / or structures at the surgical site 2325 can be visualized by an image sensor 135. Figure 2 The imaging system 142 receives data to visualize tissue and / or structures; these tissues and / or structures may be visible (e.g., located on the surface of surgical site 2325) or obscured (e.g., located below other tissues and / or structures at surgical site 2325). In this example, the imaging system 142 ( Figure 2 The imaging system 142 can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissues whose spectral characteristics do not conform to known or expected spectral characteristics) based on spectral features characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials of each of the different tissue / structure types. The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system 142, such as imaging system display 146. Figure 2 ), Main display 2119 ( Figure 18 ), non-sterile display 2109 ( Figure 18 ), Hub Display 2215 ( Figure 19), Device / Instrument Display 2237 ( Figure 19 )wait.

[0191] Furthermore, the imaging system 142 can be configured to customize or update the visualization of the displayed surgical site based on the identified tissue and / or structure type. For example, the imaging system 142 can display on a display screen (e.g., monitor 146) an edge 2330a associated with the tumor 2332 being visualized. The edge 2330a can indicate the area or amount of tissue that should be removed to ensure complete removal of the tumor 2332. Control system 133 ( Figure 2 The system 142 can be configured to control or update the size of edge 2330a based on tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified multiple anomalies 2338 within the field of view (FOV). Therefore, the control system 133 can adjust the displayed edge 2330a to a first updated edge 2330b, which has a sufficient size to cover the anomalies 2338. Furthermore, the imaging system 142 has also identified an artery 2334 (as shown by the highlighted area 2336 of artery 2334) that partially overlaps with the initially displayed edge 2330a. Therefore, the control system 133 can adjust the displayed edge 2330a to a second updated edge 2330c, which has a sufficient size to cover the relevant portion of artery 2334.

[0192] In addition to the above, regarding Figure 13A and 13B In addition to or in lieu of the described absorption properties, tissues and / or structures can also be imaged or characterized on EMR wavelength spectra based on their reflectance properties. For example, Figure 13C-13E Various graphs showing the reflectance of different types of tissues or structures at different EMR wavelengths are presented. Figure 13C This is a graphical representation of the ureteral features relative to the obscuring material 1050. Figure 13D This is a graphical representation of the illustrative arterial features relative to the obscuration 1052. Figure 13E It is a graphical representation of an exemplary neural feature relative to an obscuring object 1054. Figure 13C-13E The curves in the graphs represent the reflectance of specific structures (ureters, arteries, and nerves) relative to fat, lung tissue, and blood at corresponding wavelengths as a function of wavelength (nm). These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may have corresponding detectable reflective features that would allow for the identification and visualization of tissues and / or structures.

[0193] In various scenarios, selected wavelengths for spectral imaging can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site (i.e., “selective spectral” imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized, enabling information to be acquired in real-time or near real-time and utilized during surgery. In various scenarios, the wavelength can be selected by the clinician or by control circuitry based on clinician input. In some cases, the wavelength can be selected based on, for example, machine learning and / or large datasets accessible to the control circuitry via the cloud.

[0194] The aforementioned application of spectral imaging to tissue can be used during surgery to measure the distance between a waveform transmitter and critical structures obscured by tissue. In one aspect of this disclosure, see now. Figure 14 and Figure 15 The diagram illustrates a time-of-flight sensor system 1104 utilizing waveforms 1124 and 1125. In some cases, the time-of-flight sensor system 1104 can be integrated into a surgical visualization system 100. Figure 1 The time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. A transmitted wave 1124 extends from the transmitter 1106 to a critical structure 1101, and a received wave 1125 is reflected back from the critical structure 1101 by the receiver 1108. The surgical device 1102 is positioned through a cannula 1110 extending into a lumen 1107 in the patient.

[0195] Waveforms 1124 and 1125 are configured to penetrate the obscured tissue 1103. For example, the wavelengths of waveforms 1124 and 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal may be emitted from transmitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, waveforms 1124 and 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, transmitter 1106 is configured to provide binary signal on and off, such as Figure 15 As shown, for example, this binary signal can be measured by receiver 1108.

[0196] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d. Figure 14 ). Figure 14The flight time timing diagram 1130 of transmitter 1106 and receiver 1108 is in Figure 15 As shown in the figure. The delay is a function of the distance d, and the distance d is given by the following equation:

[0197]

[0198] in:

[0199] c = speed of light;

[0200] t = length of the pulse;

[0201] q11 = the charge accumulated during light emission; and

[0202] q2 = The charge accumulated when no light is emitted.

[0203] As provided in this article, the flight times of waveforms 1124 and 1125 correspond to Figure 14 The distance d in the distance. In various cases, the additional transmitter / receiver and / or the pulse signal from transmitter 1106 can be configured to transmit a non-penetrating signal. The non-penetrating tissue can be configured to determine the distance from the transmitter to the surface 1105 of the shielding tissue 1103. In various cases, the depth of the critical structure 1101 can be determined by the following formula:

[0204] d A =d w -d t .

[0205] in:

[0206] d A = Depth of critical structure 1101;

[0207] d w = Distance from transmitter 1106 to critical structure 1101 ( Figure 14 d) in; and

[0208] d t = The distance from the transmitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the shielding tissue 1103.

[0209] In one aspect of this disclosure, see now. Figure 16 The diagram illustrates a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, and 1225c. In some cases, the time-of-flight sensor system 1204 can be integrated into a surgical visualization system 100. Figure 1The time-of-flight sensor system 1204 includes a waveform transmitter 1206 and a waveform receiver 1208. The waveform transmitter 1206 is positioned on a first surgical device 1202a, and the waveform receiver 1208 is positioned on a second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their respective cannulas 1210a and 1210b, which extend into the patient's lumen 1207. Transmitted waves 1224a, 1224b, and 1224c extend from the transmitter 1206 toward the surgical site, and received waves 1225a, 1225b, and 1225c are reflected back to the receiver 1208 from various structures and / or surfaces at the surgical site.

[0210] Different emitted waves 1224a, 1224b, and 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, and 1224c can be in the visible, NIR, or SWIR wavelength spectrum. For example, visible light can be reflected from the surface 1205 of tissue 1203, and NIR and / or SWIR waveforms can be configured to penetrate the surface 1205 of tissue 1203. In various aspects, as described herein, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from emitter 1206. In various cases, waves 1224b and 1224c can be selected based on the spectral characteristics of key structures 1201a and 1201b to target key structures 1201a and 1201b within tissue 1203, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, which are incorporated herein by reference.

[0211] The emitted waves 1224a, 1224b, and 1224c can be reflected from the target material (i.e., surface 1205, the first key structure 1201a, and the second structure 1201b, respectively). The received waveforms 1225a, 1225b, and 1225c can be reflected due to... Figure 16 The distance d shown 1a d 2a d 3a d 1b d 2b d 2c And thus delayed.

[0212] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 can be independently positioned (e.g., positioned on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d can be calculated based on the known orientations of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when surgical devices 1202a and 1202b are controlled by a robot, these orientations can be known. Knowing the positions of transmitter 1206 and receiver 1208, the time it takes for the photon stream to target a tissue, and the information about that specific response received by receiver 1208 allows for the determination of distance d. 1a d 2a d 3a d 1b d 2b d 2c In one aspect, the distance to the shielded critical structures 1201a and 1201b can be triangulated using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0213] See still Figure 16 In various situations, in the view provided to the clinician, receiver 1208 can be rotated such that the centroid of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such orientation can rapidly transmit one or more relevant distances and / or perspectives regarding key structures. For example, as... Figure 16 As shown, the surgical site is displayed from a viewpoint perpendicular to the viewing plane (i.e., with blood vessels oriented in / outside the page) of the key structure 1201a. In various cases, this orientation may be the default setting; however, the view can be rotated or otherwise adjusted by the clinician. In some situations, the clinician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0214] In various cases, receiver 1208 may be mounted on a cannula or endotracheal tube (such as cannula 1210b), through which surgical device 1202b is positioned. In other cases, receiver 1208 may be mounted on a separate robotic arm whose three-dimensional orientation is known. In various cases, receiver 1208 may be mounted on a movable arm separate from the robot controlling surgical device 1202a, or may be mounted on an operating room (OR) table that can be registered with the robot's coordinate plane during surgery. In such cases, the orientation of transmitter 1206 and receiver 1208 may be able to be registered with the same coordinate plane, allowing for distance triangulation based on the output from time-of-flight sensor system 1204.

[0215] The combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS) (referred to as TOF-NIRS, which is capable of measuring time-resolved characteristic maps of NIR light with nanosecond resolution) can be seen in the article entitled “TIME-OF-FLIGHT NEAR-INFRAREDSPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY INGRAPEFRUIT” (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228) (the full text of which is incorporated herein by reference) and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0216] In various scenarios, time-of-flight spectral waveforms are configured to determine the depth of critical structures and / or the proximity of surgical devices to these structures. Furthermore, the various surgical visualization systems disclosed herein include surface mapping logic components configured to create a 3D rendering of the surface of visible tissue. In such cases, clinicians can know the proximity (or lack thereof) of surgical devices to critical structures even when visible tissue obscures them. In one scenario, the topography of the surgical site is provided on a monitor by the surface mapping logic component. If a critical structure is close to the surface of the tissue, spectral imaging can convey the orientation of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on known limitations of spectral imaging systems, the system is configured to convey that a critical structure is outside the range even when the spectral imaging system cannot detect it at all. Therefore, clinicians can continue to move surgical devices and / or manipulate tissue. When a critical structure moves into the range of the spectral imaging system, the system can identify the structure and thus convey that the structure is within range. In such cases, an alert can be provided when the structure is initially identified and / or when the structure is further moved within a predefined proximity region. In these situations, even if the spectral imaging system fails to identify a critical structure using known boundaries / ranges, it can still provide proximity information (i.e., lack of proximity) to clinicians.

[0217] The various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and to alert clinicians before accidental dissection and / or transection damage to critical structures. In various aspects, the surgical visualization systems are configured to identify one or more critical structures, such as the ureter, intestine, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve, and their branches), blood vessels (including the pulmonary artery and lobar artery and pulmonary vein and lobar vein, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including the middle colic artery, right colic artery, and ileal artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal (lower abdomen) ileal vessels, short gastric arteries, uterine arteries, median sacral vessels, and lymph nodes. Furthermore, the surgical visualization systems are configured to indicate the proximity of surgical devices to critical structures and / or alert clinicians when surgical devices are close to critical structures.

[0218] Various aspects of this disclosure provide for the identification of critical structures (e.g., the ureter, nerves, and / or blood vessels) and monitoring of instrument proximity during surgery. For example, the various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, enabling the visualization of critical structures, for example, below the surface of tissue (e.g., 1.0 cm to 1.5 cm below the surface of tissue). In other cases, the surgical visualization system may identify structures less than 1.0 cm or greater than 1.5 cm below the surface of tissue. For example, a surgical visualization system that can identify structures, for example, within 0.2 mm of the surface, can be valuable if the structure would otherwise not be visible due to depth. In various aspects, the surgical visualization system may, for example, utilize a virtual depiction of the critical structure as a visible white light image superimposed on the surface of visible tissue to enhance the clinician's view. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal end of a surgical instrument and may provide proximity alerts, for example, when the distal end of the surgical instrument moves within a specific range of a critical structure (e.g., within 1.0 cm of the critical structure).

[0219] The various surgical visualization systems disclosed herein can identify when anatomy is too close to critical structures. Anatomy may be “too close” to a critical structure based on temperature (i.e., too hot near a critical structure where there is a risk of damage / heating / melting) and / or tension (i.e., too much tension near a critical structure where there is a risk of damage / tearing / traction). For example, such surgical visualization systems can be beneficial for anatomy around blood vessels when skeletalizing them before ligation. In various cases, a thermal imaging camera can be used to read the heat at the surgical site and provide warnings to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the temperature of the tool is above a predefined threshold (e.g., 120℉), a warning can be provided to the clinician at a first distance (e.g., 10 mm), and if the temperature of the tool is less than or equal to the predefined threshold, a warning can be provided at a second distance (e.g., 5 mm). The predefined thresholds and / or warning distances can be default settings and / or programmable by the clinician. Alternatively or concurrently, proximity warnings may be associated with thermal measurements taken by the tool itself, such as thermocouples that measure heat in the distal jaws of a monopolar or bipolar dissecter or vascular occluder.

[0220] The various surgical visualization systems disclosed herein provide sufficient sensitivity regarding critical structures and specificities, enabling clinicians to confidently perform rapid yet safe dissections based on standards of care and / or device safety data. The systems can operate in real-time during surgery with minimal or no risk of ionizing radiation to the patient or clinician in all cases. Conversely, during fluoroscopy, patients and clinicians may be exposed to ionizing radiation via, for example, X-ray beams used for real-time observation of anatomical structures.

[0221] When the path of a surgical device is controlled by a robot, the various surgical visualization systems disclosed herein can be configured to detect and identify, for example, one or more key structures of a desired type in the forward path of the surgical device. Alternatively, the surgical visualization system can be configured to detect and identify, for example, one or more key structures of a certain type in the region surrounding the surgical device and / or in multiple planes / dimensions.

[0222] The various surgical visualization systems disclosed herein are easy to operate and / or interpret. Furthermore, these systems can be combined with "overwrite" features that allow clinicians to override default settings and / or operations. For example, clinicians may selectively disable warnings from the surgical visualization system and / or move closer to the critical structure than suggested by the system when the risk to a critical structure is less than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue may be greater than the risk of damaging the critical structure).

[0223] The various surgical visualization systems disclosed herein can be integrated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, the specific implementation of a surgical visualization system may not change how surgery is performed. Furthermore, surgical visualization systems may be more economical compared to the cost of accidental transection. Data suggests that reduced accidental damage to critical structures can drive incremental compensation.

[0224] The various surgical visualization systems disclosed in this article can operate in real-time or near real-time and far in advance, enabling clinicians to anticipate critical structures. For example, surgical visualization systems can provide sufficient time to "slow down, assess, and avoid" in order to maximize the efficiency of surgical procedures.

[0225] The various surgical visualization systems disclosed herein may not require the injection of contrast agents or dyes into the tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other cases, contrast agents can be injected into the appropriate tissue layers more easily than with other visualization systems. For instance, the time between contrast agent injection and visualization of key structures may be less than two hours.

[0226] The various surgical visualization systems disclosed herein can be correlated with clinical data and / or device data. For example, the data can provide information about how close a power-enabled surgical device (or other potentially damaging device) should be to the boundary of tissue the surgeon does not want to damage. Any data modules that interact with the surgical visualization systems disclosed herein can be provided integrally with or separately from the robot, enabling their use in conjunction with stand-alone surgical devices in, for example, open or laparoscopic surgeries. In various cases, the surgical visualization systems can be compatible with robotic surgical systems. For example, visualized images / information can be displayed on the robot's control console.

[0227] In various situations, clinicians may not know the location of critical structures relative to surgical instruments. For example, when a critical structure is embedded in tissue, the clinician may be unable to determine its location. In some cases, clinicians may want to keep surgical devices outside the azimuth range surrounding the critical structure and / or away from visible tissue covering or concealing it. When the location of the concealed critical structure is unknown, clinicians may risk moving too close to the critical structure, potentially causing unintentional trauma and / or excessive energy, heat, and / or tension on the anatomy and / or vicinity of the critical structure. Alternatively, clinicians may keep too far from the suspected location of the critical structure and risk influencing the tissue in a less than ideal position to attempt to avoid the critical structure.

[0228] This invention provides a surgical visualization system that presents surgical device tracking relative to one or more key structures. For example, the surgical visualization system can track the proximity of a surgical device to a key structure. Such tracking can occur intraoperatively, in real-time, and / or near real-time. In various cases, the tracking data can be provided to the clinician via a display screen (e.g., a monitor) of an imaging system.

[0229] In one aspect of this disclosure, a surgical visualization system includes: a surgical device including an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect an embedded structure and the structured light pattern on the visible surface; and control circuitry in signal communication with the camera and the imaging system, wherein the control circuitry is configured to determine a distance from the surgical device to the embedded structure and to provide a signal indicative of that distance to the imaging system. For example, the distance can be determined by calculating the distance from the camera to a critical structure illuminated using fluorescence fluoroscopy and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses (e.g., a left lens and a right lens) of the camera. For example, the distance from the surgical device to the critical structure can be triangulated based on the known orientation of the surgical device and the camera. Alternative devices for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue superimposed / covering the embedded critical structure. For example, surgical visualization systems can identify and enhance the view of hidden critical structures by drawing a schematic diagram (such as a line on the surface of visible tissue) over visible structures. Surgical visualization systems can also determine the distance to the enhancement line on the visible tissue.

[0230] By providing clinicians with up-to-date information on the proximity of surgical devices to concealed critical structures and / or visible structures, as disclosed in this article through various surgical visualization systems, clinicians can make more informed decisions regarding the placement of surgical devices relative to concealed critical structures. For example, clinicians can view the distance between the surgical device and the critical structure in real time / intraoperatively, and in some cases, the imaging system can provide alerts and / or warnings when the surgical device moves into a predefined proximity and / or area of ​​the critical structure. In some cases, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly zone" near the critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more). In such cases, clinicians can maintain momentum throughout the surgical procedure without needing to monitor the suspected location of the critical structure and the proximity of the surgical device to it. Therefore, some surgical procedures can be performed more quickly with fewer pauses / interruptions and / or with improved accuracy and / or certainty. In one respect, surgical visualization systems can be used to detect tissue variability, such as the variability of tissues within organs, to distinguish between tumor / cancer / unhealthy tissue and healthy tissue. Such surgical visualization systems can maximize the removal of unhealthy tissue while minimizing the removal of healthy tissue.

[0231] Surgical hub system

[0232] The various visualization or imaging systems described in this article can be incorporated into surgical hub systems, such as by combining... Figure 17-19 It is shown and described in further detail below.

[0233] See Figure 17 The computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. In one example, as... Figure 17 As shown, the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and handheld intelligent surgical instruments 2112, which are configured to communicate with each other and / or with a hub 2106. In some aspects, the surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to one.

[0234] Figure 18 An example of a surgical system 2102 for performing surgery on a patient lying supine on an operating table 2114 in an operating room 2116 is shown. A robotic system 2110 is used as part of the surgical system 2102 during the surgery. The robotic system 2110 includes a surgeon's console 2118, a patient-side trolley 2120 (surgical robot), and a surgical robot hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side trolley 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision within the patient's body. Images of the surgical site can be obtained via a medical imaging device 2124, which can be manipulated by the patient-side trolley 2120 to orient the imaging device 2124. The robot hub 2122 can be used to process the images of the surgical site for subsequent display to the surgeon via the surgeon's console 2118.

[0235] Other types of robotic systems can be readily adapted for use with surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0236] Various examples of cloud-based analytics performed by Cloud 2104 and applicable to this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0237] In various respects, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0238] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate multiple portions of the surgical site. One or more image sensors may receive light reflected or refracted from the surgical site, including light reflected or refracted from tissue and / or surgical instruments.

[0239] One or more illumination sources can be configured to radiate electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or emission spectrum) is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by it) and can be called visible light or simple light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0240] The invisible spectrum (i.e., the non-luminescent spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0241] In various respects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, colonoscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0242] In one aspect, imaging devices employ multispectral monitoring to discern morphology and underlying structure. A multispectral image is an image that captures image data across a specific wavelength range of the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated herein by reference. After completing a surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site.

[0243] It goes without saying that rigorous sterilization of the operating room and surgical equipment is required during any surgical procedure. The stringent hygienic and sterilization conditions required in the “surgical room” (i.e., operating room or treatment room) necessitate the highest possible sterility of all medical devices and equipment. Part of this sterilization process requires the sterilization of any material that comes into contact with the patient or penetrates the sterile area, including the imaging device 2124 and its attachments and components. It should be understood that a sterile area can be considered a designated area deemed free of microorganisms, such as within a tray or sterile towel, or can be considered the area around the patient prepared for surgical procedures. A sterile area may include properly dressed scrubbed team members, as well as all equipment and fixtures within that area.

[0244] In various aspects, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays, strategically arranged relative to a sterile area, such as... Figure 18 As shown in the figure. In one aspect, visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of visualization system 2108 are described in various U.S. patent applications, which are incorporated herein by reference.

[0245] like Figure 18 As shown, the main display 2119 is positioned within the sterile area for visibility to the operator at the operating table 2114. Furthermore, a visualization tower 21121 is positioned outside the sterile area. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 positioned opposite each other. A visualization system 2108, guided by a hub 2106, is configured to utilize displays 2107, 2109, and 2119 to coordinate information flow to operators both inside and outside the sterile area. For example, the hub 2106 allows the visualization system 2108 to display snapshots of the surgical site recorded by the imaging device 2124 on the non-sterile displays 2107 or 2109 while maintaining a real-time feed of the surgical site on the main display 2119. The snapshots on the non-sterile displays 2107 or 2109 may allow a non-sterile operator to perform diagnostic steps related to the surgical procedure, for example.

[0246] In one aspect, hub 2106 is further configured to route diagnostic inputs or feedback entered by a non-sterile operator at visualization tower 21121 to a main display 2119 within a sterile area, which can be viewed by a sterile operator at a workbench. In one example, the input may be a modified form of a snapshot displayed on non-sterile displays 2107 or 2109, which can be routed to the main display 2119 via hub 2106.

[0247] See Figure 18 Surgical instrument 2112 is used in surgical procedures as part of surgical system 2102. Hub 2106 is also configured to coordinate information flow to the display of surgical instrument 2112, as described in various U.S. patent applications, which are incorporated herein by reference. Diagnostic input or feedback entered by a non-sterile operator at visualization tower 21121 can be routed by hub 2106 to surgical instrument display 2115 within a sterile area, where the operator of surgical instrument 2112 can observe the input or feedback. Exemplary surgical instruments suitable for surgical system 2102 are described in various U.S. patent applications, which are incorporated herein by reference.

[0248] Figure 19 An interactive surgical system 2200 implemented by a computer is illustrated. The interactive surgical system 2200 implemented by a computer is similar in many respects to the interactive surgical system 2100 implemented by a computer. The surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204, which may include a remote server 2213. In one aspect, the interactive surgical system 2200 implemented by a computer includes a surgical hub 2236 that connects to multiple operating room devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively connecting the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. Figure 19 As illustrated in the example, the surgical hub 2236 is coupled to an imaging module 2238 (which is coupled to an endoscope 2239), a generator module 2240 coupled to an energy device 2421, a fume extractor module 2226, a suction / rinsing module 2228, a communication module 2230, a processor module 2232, a storage array 2234, an intelligent device / instrument 2235 optionally coupled to a display 2237, and a non-contact sensor module 2242. Operating room equipment is coupled to cloud computing resources and data storage via the surgical hub 2236. A robotic hub 2222 can also be connected to the surgical hub 2236 and cloud computing resources. Devices / instruments 2235, visualization systems 2209, etc., can be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols, as described herein. The surgical hub 2236 can be coupled to a hub display 2215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display, and / or other visualization system 208. The hub display can also combine and overlay images to display data received from devices connected to a modular control tower.

[0249] Situational awareness

[0250] The various visualization systems or aspects thereof described herein can be used as part of a situational awareness system, which can be comprised of surgical hubs 2106, 2236 ( Figure 17-19 The situational awareness system can use this situational data to implement or perform surgical procedures. Specifically, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and movement), tissues, structures, users, and other elements located in the surgical field or operating room can provide contextual data. The situational awareness system can then use this contextual data to infer the type of surgical procedure being performed or its steps, the type of tissue and / or structure the surgeon is manipulating, etc. The situational awareness system can then use this contextual data to provide alerts to the user, suggest subsequent steps or actions, prepare surgical devices for use (e.g., activate an electrosurgical generator as expected in a subsequent step of the surgical procedure using an electrosurgical instrument), intelligently control surgical instruments (e.g., customize surgical instrument operating parameters based on each patient's specific health condition), etc.

[0251] While a “smart” device that includes control algorithms responding to sensed data can be an improvement over a “dumb” device that operates without considering sensed data, some sensed data can be incomplete or uncertain when considered in isolation—that is, without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context (e.g., knowing the type of tissue being operated on or the type of surgery being performed), the control algorithm may incorrectly or suboptimally control the modular device given specific, context-free sensed data. Modular devices can include any surgical device that can be controlled by a situational awareness system, such as visualization system devices (e.g., cameras or displays), surgical instruments (e.g., ultrasound surgical instruments, electrosurgical instruments, or surgical sutures), and other surgical devices (e.g., fumigators). For example, the optimal approach for a control algorithm to control surgical instruments in response to specific sensed parameters can vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and therefore respond differently to actions taken by surgical instruments. Therefore, it may be expected that surgical instruments will act differently even when sensing the same measurement for a specific parameter. As a specific example, the optimal way to control surgical suturing and cutting instruments in response to unexpectedly high forces sensed by the instrument for closing its end effector will vary depending on whether the tissue type is prone to tearing or resistant to tearing. For easily tearable tissues (such as lung tissue), the instrument's control algorithm will optimally decrease the motor speed gradually in response to unexpectedly high forces for closure, thereby avoiding tissue tearing. For resistant tissues (such as stomach tissue), the instrument's control algorithm will optimally increase the motor speed gradually in response to unexpectedly high forces for closure, thereby ensuring that the end effector is properly clamped onto the tissue. In cases where it is unknown whether lung or stomach tissue has been clamped, the control algorithm may make a suboptimal decision.

[0252] One solution utilizes a surgical hub comprising a system configured to derive information about a surgical procedure being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about a surgical procedure from the received data, and then control modular devices paired with the surgical hub based on the inferred context of the surgical procedure. Figure 20A diagram of a situational awareness surgical system 2400 according to at least one aspect of this disclosure is shown. In some examples, the data source 2426 includes, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database containing patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0253] The surgical hub 2404 (which may be similar to hub 106 in many respects) may be configured to derive surgical context information from the data, for example, based on a specific combination of received data or a specific order in which data is received from data source 2426. The context information inferred from the received data may include, for example, the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity of the object of the procedure. This ability of the surgical hub 2404 to derive or infer surgical context information from the received data may be referred to as “situational awareness.” In one example, the surgical hub 2404 may be incorporated into a situational awareness system, which is the hardware and / or programming associated with the surgical hub 2404 for deriving surgical context information from the received data.

[0254] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data source 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to associate various inputs (e.g., data from the database 2422, patient monitoring device 2424, and / or modular device 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another example, the situational awareness system may include a lookup table that stores pre-represented contextual information about the surgical procedure associated with one or more inputs (or ranges of inputs) corresponding to the contextual information. In response to a query using one or more inputs, the lookup table can return the corresponding contextual information used by the situational awareness system to control the modular device 2402. In one example, the contextual information received by the situational awareness system of the surgical hub 2404 is associated with a specific control adjustment or a set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes additional machine learning systems, lookup tables, or other such systems that generate or retrieve one or more control adjustments for one or more modular devices 2402 when provided with contextual information as input.

[0255] The surgical hub 2404, incorporating a situational awareness system, provides numerous benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or data utilization during surgical procedures. Returning to the previous example, the situational awareness surgical hub 2404 can determine the type of tissue being operated on; therefore, when an unexpectedly high force is detected on the end effector of the surgical instrument used for closure, the situational awareness surgical hub 2404 can correctly adjust the motor speed of the surgical instrument according to the tissue type by gradually increasing or decreasing it.

[0256] As another example, the type of tissue being operated on can influence the adjustment of the compression rate and load threshold of surgical suture and cutting instruments for specific tissue gap measurements. The situation-aware surgical hub 2404 can infer whether the surgery being performed is thoracic or abdominal, allowing it to determine whether the tissue held by the end effector of the surgical suture and cutting instruments is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical suture and cutting instruments according to the tissue type.

[0257] As another example, the type of body cavity operated on during a blow-through procedure can affect the function of the smoke extractor. The situation-aware surgical hub 2404 can determine whether the surgical site is under pressure (by determining that a surgical procedure is being performed using blow-through) and determine the type of surgery. Since a type of surgery is typically performed within a specific body cavity, the surgical hub 2404 can then appropriately control the motor speed of the smoke extractor for the body cavity in which the procedure is being performed. Therefore, the situation-aware surgical hub 2404 can provide consistent smoke extraction for both thoracic and abdominal surgeries.

[0258] As yet another example, the type of procedure being performed can influence the optimal energy level for operating ultrasound surgical instruments or radiofrequency (RF) electrosurgical instruments. For instance, arthroscopic procedures require higher energy levels because the end effectors of the ultrasound surgical instruments or RF electrosurgical instruments are immersed in fluid. A situational-aware surgical hub 2404 can determine whether a surgical procedure is arthroscopic. The surgical hub 2404 can then adjust the RF power level or ultrasound amplitude (i.e., the “energy level”) of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can influence the optimal energy level for operating ultrasound surgical instruments or RF electrosurgical instruments. The situational-aware surgical hub 2404 can determine the type of surgical procedure being performed and then tailor the energy levels of the ultrasound surgical instruments or RF electrosurgical instruments separately based on the expected tissue profile of that surgical procedure. Furthermore, the situational-aware surgical hub 2404 can be configured to adjust the energy levels of the ultrasound surgical instruments or RF electrosurgical instruments throughout the entire surgical procedure, rather than just on a per-procedure basis. The situational awareness surgical hub 2404 can determine the steps of a surgical procedure being performed or to be performed subsequently, and then update the control algorithms for the generator and / or ultrasound surgical instruments or RF electrosurgical instruments to set the energy level to a value suitable for the expected tissue type based on the surgical step.

[0259] As another example, data can be extracted from additional data source 2426 to improve the conclusions drawn by surgical hub 2404 from one data source 2426. The situational-aware surgical hub 2404 can augment the data received from modular device 2402 with background information about the surgical procedure already constructed from other data sources 2426. For example, the situational-aware surgical hub 2404 can be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) based on video or image data received from a medical imaging device. However, in some cases, the video or image data may be indeterminate. Therefore, in one example, surgical hub 2404 can be further configured to compare physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively connected to surgical hub 2404) with visual or image data of hemostasis (e.g., from medical imaging device 124 communicatively coupled to surgical hub 2404). Figure 2 The comparison is used to determine the integrity of sutures or tissue welds. In other words, the situational awareness system of the surgical hub 2404 can take physiological measurement data into account to provide additional context when analyzing visualization data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0260] Another benefit includes the proactive and automated control of the paired modular device 2402 based on specific steps of the surgical procedure being performed, reducing the number of times medical personnel need to interact with or control the surgical system 2400 during the surgical procedure. For example, if the situation-aware surgical hub 2404 determines that a subsequent step of the procedure requires the use of an RF electrosurgical instrument, it can proactively activate a generator connected to that instrument. Proactive activation of the power source allows the instrument to be ready for use immediately after the previous steps of the procedure have been completed.

[0261] As another example, the situation-aware surgical hub 2404 can determine whether the current or subsequent steps of the surgical procedure require a different view or magnification on the display based on one or more features(s) that the surgeon expects to view at the surgical site. The surgical hub 2404 can then proactively change the displayed view accordingly (e.g., provided by a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0262] As yet another example, the situation-aware surgical hub 2404 can determine which step of a surgical procedure is being performed or will be performed subsequently, and whether that step of the procedure requires specific data or comparisons between data. The surgical hub 2404 can be configured to automatically invoke a data screen based on the step of the surgical procedure being performed, without waiting for the surgeon to request that specific information.

[0263] Another benefit includes error detection during the setup of a surgical procedure or during the procedure itself. For example, a situational-aware surgical hub 2404 can determine whether the operating room is correctly or optimally set up for a surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgery being performed, retrieve (e.g., from memory) a corresponding list, product location, or setup requirement, and then compare the current operating room layout with a standard layout determined by the surgical hub 2404 for that type of surgery. In one example, the surgical hub 2404 can be configured to compare a list of items for the procedure (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 2404 with a suggested or anticipated list of items and / or devices for a given surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating the absence of a specific modular device 2402, patient monitoring device 2424, and / or other surgical items if any discontinuities exist between the lists. In one example, the surgical hub 2404 may be configured to determine, for example, the relative distance or position of the modular device 2402 and the patient monitoring device 2424 via a proximity sensor. The surgical hub 2404 may compare the relative position of the devices with a suggested or anticipated layout for a particular surgical procedure. The surgical hub 2404 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the suggested layout if any discontinuity exists between the layouts.

[0264] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from the intended procedure during a surgical operation. For instance, the surgical hub 2404 can be configured to determine the type of surgery being performed, retrieve (e.g., from memory) a corresponding list of steps or the order of equipment used, and then compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment determined by the surgical hub 2404 for that type of surgery. In one example, the surgical hub 2404 can be configured to provide an alarm indicating that an unexpected action is being performed or an unexpected device is being used at a specific step in the surgical procedure.

[0265] Overall, the situational awareness system used in the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (such as adjusting for different tissue types) and verifying actions during surgery. The situational awareness system also enhances the efficiency of surgeons performing surgical procedures by automatically suggesting the next step, providing data, and adjusting displays and other modular devices 2402 in the operating room, based on the specific context of the surgery.

[0266] See now Figure 21 It shows depictions of hubs such as surgical hubs 106 or 206 ( Figures 1 to 11 The situational awareness timeline 2500 is an illustrative surgical procedure and background information that surgical hubs 106 and 206 can derive from data received from the data source at each step of the surgical procedure. Timeline 2500 depicts the typical steps that nurses, surgeons, and other medical personnel will take during a segmentectomy, from setting up the operating room to transferring the patient to the postoperative recovery room.

[0267] Situational awareness surgical hubs 106 and 206 receive data from data sources throughout the surgical procedure, including data generated each time medical personnel utilize the modular devices paired with the surgical hubs 106 and 206. The surgical hubs 106 and 206 can receive this data from paired modular devices and other data sources, and continuously derive inferences about the ongoing surgery (i.e., background information) as new data is received, such as which step of the surgery is being performed at any given time. The situational awareness system of the surgical hubs 106 and 206 is capable, for example, recording data related to the process used to generate reports, verifying the steps being taken by medical personnel, providing data or cues that may be related to specific procedural steps (e.g., via a display screen), adjusting modular devices based on the background (e.g., activating monitors, adjusting the field of view (FOV) of medical imaging devices, or changing the energy level of ultrasound surgical instruments or RF electrosurgical instruments), and taking any other such actions described above.

[0268] As a first step 2502 in this exemplary procedure, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, surgical hubs 106 and 206 determine that the procedure to be performed is a thoracic surgery.

[0269] In the second step 2504, the staff scans the medical supplies to be brought in for the surgery. Surgical hubs 106 and 206 cross-reference the scanned supplies with a list of supplies used in various types of surgery and confirm that the supplied mixture corresponds to a thoracic surgery. Additionally, surgical hubs 106 and 206 can also determine if the surgery is not a wedge surgery (because the brought-in supplies lack certain supplies required for a thoracic wedge surgery, or are otherwise not corresponding to a thoracic wedge surgery).

[0270] In the third step 2506, medical personnel scan the patient band via a scanner communicatively connected to the surgical hubs 106 and 206. The surgical hubs 106 and 206 can then identify the patient based on the scanned data.

[0271] In step 4, 2508, medical staff activate assistive devices. The assistive devices used can vary depending on the type of surgery and the techniques the surgeon intends to use, but in this exemplary case, they include a fumigator, a blower, and a medical imaging device. Upon activation, as part of its initialization process, the assistive devices, as modular devices, can automatically pair with surgical hubs 106, 206 located in a specific vicinity of the modular device. Surgical hubs 106, 206 can then derive background information about the surgery by detecting the type of modular device paired with them during this preoperative or initialization phase. In this specific example, surgical hubs 106, 206 determine that the surgery is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies used in the surgery, and the type of modular device connected to the hub, surgical hubs 106, 206 can generally infer the specific procedure the surgical team will perform. Once the surgical hubs 106 and 206 know what specific surgery is being performed, they can retrieve the steps of the surgery from memory or the cloud and then cross-reference them with data subsequently received from connected data sources (e.g., modular devices and patient monitoring devices) to infer which step of the surgical procedure the surgical team is performing.

[0272] In step 5, 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices are compatible with surgical hubs 106 and 206. When surgical hubs 106 and 206 begin receiving data from the patient monitoring devices, they thus confirm that the patient is in the operating room.

[0273] Step 6, 2512: Medical personnel induce anesthesia in the patient. Surgical hubs 106 and 206 can infer that the patient is under anesthesia based on data from the modular device and / or patient monitoring devices (including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof). Upon completion of step 6, 2512, the preoperative portion of the lung segmental resection surgery is completed, and the surgical portion begins.

[0274] Step 7, 2514: Fold the patient's lung being operated on (while simultaneously switching ventilation to the contralateral lung). For example, surgical hubs 106 and 206 can infer from ventilator data that the patient's lung has collapsed. Surgical hubs 106 and 206 can infer that the surgical portion of the procedure has begun because they can compare the detection of lung collapse with the expected steps of the procedure (which can be previously accessed or retrieved), thereby determining that collapsing the lung is the first surgical step in that particular procedure.

[0275] Step 8, 2516: Insert a medical imaging device (e.g., an endoscope) and activate the video from the medical imaging device. Surgical hubs 106 and 206 receive data (i.e., video or image data) from the medical imaging device via their connection. After receiving the data, surgical hubs 106 and 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, surgical hubs 106 and 206 can determine that the specific procedure being performed is a segmental resection, not a lobectomy (note that wedge resection has been excluded based on data received by surgical hubs 106 and 206 at step 2504 of the procedure). From medical imaging device 124 ( Figure 2The data can be used to determine contextual information relevant to the type of surgery being performed in a variety of different ways, including by determining the angle of the visualization orientation of the medical imaging device relative to the patient's anatomy, monitoring the number of medical imaging devices used (i.e., those activated and paired with surgical hubs 106, 206), and monitoring the type of visualization device used. For example, a technique for performing a VATS lobectomy places the camera in the lower anterior corner of the patient's thoracic cavity above the diaphragm, while a technique for performing a VATS segmental resection places the camera in the anterior intercostal position relative to the segmental fissure. For example, the situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to identify the positioning of the medical imaging device based on the visualization of the patient's anatomy. As another example, a technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmental resection utilizes multiple cameras. As yet another example, a technique for performing a VATS segmental resection utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure not used in VATS lobectomy. By tracking any or all of this data from medical imaging devices, surgical hubs 106, 206 can thus determine the specific type of surgery being performed and / or the technique used for a particular type of surgery.

[0276] Step 9, 2518, involves the surgical team initiating the anatomical step of the procedure. Surgical hubs 106 and 206 can infer that the surgeon is dissecting to access the patient's lungs because they receive data from an RF generator or ultrasound generator indicating the firing of an energy device. Surgical hubs 106 and 206 can cross-reference the received data with the surgical procedure's retrieval steps to determine which point in the process (i.e., after the previously discussed surgical steps have been completed) corresponds to the anatomical step. In some cases, the energy device may be a power tool for the robotic arm mounted to the robotic surgical system.

[0277] Step 10, 2520: The surgical team continues with the ligation step of the surgery. Surgical hubs 106 and 206 can infer that the surgeon is ligating arteries and veins because they receive data from the surgical suture and cutting instruments indicating that the instruments are being fired. Similar to previous steps, surgical hubs 106 and 206 can deduce this inference by cross-referencing the data received from the surgical suture and cutting instruments with the retrieval steps in this process. In some cases, the surgical instruments may be surgical tools mounted on the robotic arm of a robotic surgical system.

[0278] Step 11, 2522: Performing the segmental resection portion of the surgery. Surgical hubs 106 and 206 can infer that the surgeon is transecting soft tissue based on data from surgical suture and cutting instruments (including data from their chambers). Chamber data may correspond to, for example, the size or type of staples fired by the instruments. Since different types of staples are used for different types of tissue, the chamber data can indicate the type of tissue being sutured and / or transected. In this case, the type of staple fired is used for soft tissue (or other similar tissue type), which allows surgical hubs 106 and 206 to infer that the segmental resection portion of the surgery is underway.

[0279] In step 12, 2524, the node dissection step is performed. Surgical hubs 106 and 206 can infer that the surgical team is dissecting a node and performing a leak test based on data received from the generator indicating that an RF or ultrasound instrument is being fired. For this particular surgery, the RF or ultrasound instrument used after transverse soft tissue incision corresponds to the node dissection step, which allows surgical hubs 106 and 206 to make such inferences. It should be noted that surgeons periodically switch between surgical suture / cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the specific steps in the surgery, as different instruments are better suited to specific tasks. Therefore, a specific sequence in which suture / cutting instruments and surgical energy instruments are used can indicate the steps of the surgery being performed by the surgeon. Furthermore, in some cases, robotic tools may be used for one or more steps in the surgery, and / or handheld surgical instruments may be used for one or more steps in the surgery. One or more surgeons may, for example, alternate between robotic tools and handheld surgical instruments and / or use the devices simultaneously. Upon completion of step 12, 2524, the incision is closed and the postoperative portion of the surgery begins.

[0280] Step 13, 2526: Reverse anesthesia of the patient. For example, surgical hubs 106 and 206 can infer that the patient is waking up from anesthesia based on, for example, ventilator data (i.e., the patient's respiratory rate begins to increase).

[0281] Finally, step fourteen, 2528, involves medical personnel removing various patient monitoring devices from the patient. Therefore, when the hub loses EKG, BP, and other data from the patient monitoring devices, surgical hubs 2106 and 2236 can infer that the patient is being transferred to the recovery room. As can be seen from the description of this exemplary procedure, surgical hubs 2106 and 2236 can determine or infer when each step of a given surgical procedure occurs based on data received from various data sources communicatively coupled to the surgical hubs 2106 and 2236.

[0282] Situational awareness is further described in various U.S. patent applications, which are incorporated herein by reference, and this disclosure is also incorporated herein by reference. In certain circumstances, the operation of a robotic surgical system (including, for example, the various robotic surgical systems disclosed herein) may be based on the situational awareness of hubs 2106 and 2236 and / or feedback from their components and / or from cloud 2104 ( Figure 17 It uses information to control [the system].

[0283] Figure 22 This is a logic flowchart of process 4000 according to at least one aspect of the present disclosure, which depicts a control procedure or logical configuration for associating visualization data with instrument data. Process 4000 is typically performed during surgical procedures and includes: receiving or exporting 4001 a first dataset, i.e., visualization data, indicating the visual aspect of surgical instruments relative to a surgical field of view from a surgical visualization system; receiving or exporting 4002 a second dataset, i.e., instrument data, indicating the functional aspect of surgical instruments from surgical instruments; and associating the first dataset with the second dataset 4003.

[0284] In at least one example, the association between visualization data and instrumentation data is achieved by developing a composite dataset from visualization data and instrumentation data. Process 4000 may further include comparing the composite dataset with another composite dataset, which may be received from an external source and / or derived from a previously collected composite dataset. In at least one example, process 4000 includes displaying a comparison of the two composite datasets, as described in more detail below.

[0285] The visualization data of process 4000 may indicate the visual aspect of the end effector of the surgical instrument relative to tissue in the surgical field of vision. Alternatively or additionally, the visualization data may indicate the visual aspect of tissue treated by the end effector of the surgical instrument. In at least one example, the visualization data represents one or more positions of the end effector or its components relative to tissue in the surgical field of vision. Alternatively or additionally, the visualization data may represent one or more movements of the end effector or its components relative to tissue in the surgical field of vision. In at least one example, the visualization data represents one or more changes in the shape, size, and / or color of tissue treated by the end effector of the surgical instrument.

[0286] In all aspects, visualization data is derived from surgical visualization systems (e.g., visualization systems 100, 160, 500, 2108). Visualization data can be derived from various measurements, readings, and / or any other suitable parameters monitored and / or captured by the surgical visualization system, such as in combination with... Figures 1 to 18To describe in more detail. In various examples, the visualization data indicates one or more visual aspects of tissue in the surgical field of view and / or one or more visual aspects of surgical instruments relative to tissue in the surgical field of view. In some examples, the visualization data represents or identifies the end effector of a surgical instrument relative to key structures in the surgical field of view (e.g., Figure 1 The location and / or movement of the key structure 101 in the image. In some examples, the visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data calculated by surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, and 141.

[0287] In at least one example, the visualization data is derived from tissue identification and geometric surface mapping performed by the visualization system 100 in conjunction with the distance sensor system 104, such as in conjunction with Figure 1 A more detailed description follows. In at least one example, the visualization data is derived from measurements, readings, or any other sensor data captured by the imaging device 120. (As in conjunction with...) Figure 1 The imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate spectral cubes of images based on molecular responses to different wavelengths.

[0288] Alternatively or concurrently, the visualization data may be derived from measurements, readings, or any suitable sensor data captured by any suitable imaging device, including, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In at least one example, the visualization data is derived from a visualization system 160, which includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms, such as in combination with... Figures 3 to 4 and Figure 13 to Figure 16 A more detailed description follows. In yet another example, the visualization data is derived from a visualization system that includes a 3D camera and associated electronic processing circuitry, such as visualization system 500. In yet another example, the visualization data is derived from a structured (or patterned) light system 700, which combines... Figure 12 A more detailed description follows. The foregoing examples can be used individually or in combination to export visualizations of Process 4000 data.

[0289] The instrument data in process 4000 may indicate one or more operations of one or more internal components of a surgical instrument. In at least one example, the instrument data represents one or more operating parameters of the internal components of the surgical instrument. The instrument data may represent one or more positions and / or one or more movements of one or more internal components of the surgical instrument. In at least one example, the internal component is a cutting member configured to cut tissue during a firing sequence of the surgical instrument. Alternatively or additionally, the internal component may include one or more pins configured to be fired into tissue during a firing sequence of the surgical instrument.

[0290] In at least one example, the instrument data represents one or more operations of one or more components of one or more drive components (e.g., joint motion drive component, closure drive component, rotation drive component, and / or firing drive component) of a surgical instrument. In at least one example, the instrument dataset represents one or more operations of one or more drive members (e.g., joint motion drive member, closure drive member, rotation drive member, and / or firing drive member) of a surgical instrument.

[0291] Figure 23 This is a schematic diagram of an exemplary surgical instrument 4600 used with process 4000, which is similar in many respects to other surgical instruments or tools described in this disclosure (e.g., surgical instrument 2112). For the sake of brevity, this disclosure uses only handheld surgical instruments to describe various aspects of process 4000. However, this is not limiting. Such aspects of process 4000 can also be implemented using robotic surgical tools (e.g., surgical tool 2117).

[0292] Surgical instrument 4600 includes multiple motors that can be activated to perform various functions. The multiple motors of surgical instrument 4600 can be activated to cause firing motion, closing motion, and / or articulation motion in an end effector. The firing motion, closing motion, and / or articulation motion can be transmitted to the end effector of surgical instrument 4600, for example, via a shaft assembly. However, in other examples, the surgical instrument used with process 4000 can be configured to manually perform one or more of the firing motion, closing motion, and articulation motion. In at least one example, surgical instrument 4600 includes an end effector that treats tissue by deploying a staple into the tissue. In another example, surgical instrument 4600 includes an end effector that treats tissue by applying therapeutic energy to the tissue.

[0293] In some cases, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 may be operatively coupled to a firing motor drive assembly 4604, which may be configured to transmit the firing motion generated by the firing motor 4602 to an end effector, specifically for moving a firing member in the form of an I-beam, which may include a cutting member, for example. In some cases, the firing motion generated by the firing motor 4602 may cause, for example, a staple to be deployed from a staple cartridge into tissue captured by the end effector, and optionally, cause the cutting member of the I-beam to be advanced to cut the captured tissue.

[0294] In some cases, surgical instruments or tools may include a closure motor 4603. The closure motor 4603 may be operatively coupled to a closure motor drive assembly 4605, which is configured to transmit the closing motion generated by the closure motor 4603 to an end effector, specifically for displacing the closure tube to close the anvil and compress tissue between the anvil and the cartridge. The closing motion may cause, for example, the end effector to change from an open configuration to an approach configuration to capture tissue.

[0295] In some cases, surgical instruments or tools may include, for example, one or more articulated motors 4606a, 4606b. The articulated motors 4606a, 4606b may be operatively coupled to corresponding articulated motor drive assemblies 4608a, 4608b, which may be configured to transmit joint motion generated by the articulated motors 4606a, 4606b to an end effector. In some cases, the joint motion may cause the end effector to articulate relative to an axis, for example.

[0296] In some cases, surgical instruments or tools may include a control module 4610 that can be used with multiple motors of the surgical instrument 4600. Each of motors 4602, 4603, 4606a, and 4606b may include a torque sensor to measure the output torque on the motor shaft. Forces on the end effector can be sensed in any conventional manner, such as by a force sensor on the outside of the jaws or by a torque sensor for the motor used to actuate the jaws.

[0297] In various situations, such as Figure 23 As shown, control module 4610 may include motor driver 4626, which may include one or more H-bridge FETs. Motor driver 4626 may, for example, modulate the power delivered from power source 4628 to the motor coupled to control module 4610 based on input from microcontroller 4620 (“controller”). In some cases, when the motor is coupled to control module 4610, controller 4620 may be used, for example, to determine the current consumed by the motor, as described above.

[0298] In some cases, controller 4620 may include microprocessor 4622 (“processor”) and one or more non-transitory computer-readable medium or memory units 4624 (“memory”). In some cases, memory 4624 may store various program instructions that, when executed, cause processor 4622 to perform the various functions and / or calculations described herein. In some cases, one or more memory units in memory unit 4624 may be coupled to processor 4622, for example. In various cases, processor 4622 may control motor driver 4626 to control the position, direction of rotation, and / or speed of a motor coupled to control module 4610.

[0299] In some cases, one or more mechanisms and / or sensors (e.g., sensor 4630) may be configured to detect the force (closing force "FTC") applied by the jaws of the end effector of surgical instrument 4600 to tissue captured between the jaws. The FTC may be transmitted to the jaws of the end effector via the closing motor drive assembly 4605. Alternatively or additionally, sensor 4630 may be configured to detect the force (firing force "FTF") applied to the end effector via the firing motor drive assembly 4604. In various examples, sensor 4630 may be configured to sense closing actuation (e.g., motor current and FTC), firing actuation (e.g., motor current and FTF), joint movement (e.g., angular position of the end effector), and rotation of the shaft or end effector.

[0300] One or more aspects of process 4000 may be executed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4000 are executed by control circuits (e.g., Figure 2A The control circuit 4000 executes the process, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4000. Alternatively or additionally, one or more aspects of the process 4000 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The process 4000 is executed by the control circuit 420. Furthermore, the process 4000 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0301] In all aspects, the process 4000 can be implemented via a computer-based interactive surgical system 2100. Figure 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4000 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108) and may communicate with surgical instruments (e.g., surgical instruments 2112, 4600) to receive instrument data from them. Communication between the surgical instruments and the control circuitry of the visualization system may be direct communication, or instrument data may be routed to the visualization system via, for example, the surgical hub 2106. In at least one example, the control circuitry for one or more aspects of the execution process 4000 may be a component of the surgical hub 2106.

[0302] refer to Figure 24 In various examples, visualization data 4010 is associated with instrument data 4011 by developing a composite dataset 4012 from visualization data 4010 and instrument data 4011. Figure 24 A composite dataset 4012 for the current user is shown in graph 4013, which is developed from the current user's visualization data 4010 and instrument data 4011. Graph 4013 shows visualization data 4010 representing the first use cycle of the surgical instrument 4600, which involves jaw positioning, clamping, and firing of the surgical instrument 4600. Graph 4013 also depicts visualization data 4010 representing the start of the second use cycle of the surgical instrument 4600, where the jaws are repositioned for a second clamping and firing of the surgical instrument 4600. Graph 4013 further depicts the current user's instrument data 4011 associated with the clamping visualization data in the form of FTC data 4014 and with the firing visualization data in the form of FTF data 4015.

[0303] As described above, visualization data 4010 is derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can represent, for example, the distance between the end effector of the surgical instrument 4600 and a critical structure in the surgical field of view during positioning, clamping, and / or firing of the end effector of the surgical instrument 4600. In at least one example, the visualization system identifies the end effector or a component thereof in the surgical field of view, identifies the critical structure in the surgical field of view, and tracks the position of the end effector or a component thereof relative to the critical structure or relative to the tissue surrounding the critical structure. In at least one example, the visualization system identifies the jaws of the end effector in the surgical field of view, identifies the critical structure in the surgical field of view, and tracks the position of the jaws relative to the critical structure or relative to the tissue surrounding the critical structure during surgery.

[0304] In at least one example, the key structure is a tumor. To remove a tumor, surgeons typically prefer to cut tissue along a safe margin around the tumor to ensure complete removal. In such examples, visualization data 4010 may represent the distance between the jaws of the end effector and the safe margin of the tumor during the positioning, clamping, and / or firing of the surgical instrument 4600.

[0305] Process 4000 may further include comparing the current user's composite dataset 4012 with another composite dataset 4012', which may be received from an external source and / or derived from a previously collected composite dataset. A graph 4013 illustrates the comparison between the current user's composite dataset 4012 and another composite dataset 4012', which includes visualization data 4010' and instrument data 4011', including FTC data 4014' and FTF data 4015'. The comparison may be presented in real-time to the user of the surgical instrument 4600 in the form of graph 4013 or any other suitable format. Control circuitry performing one or more aspects of process 4000 may cause the comparison of the two composite datasets to be displayed on any suitable screen in the operating room (e.g., the screen of a visualization system). In at least one example, the comparison may be displayed along with real-time video of the surgical field of view captured on any suitable screen in the operating room. In at least one example, the control circuitry is configured to adjust instrument parameters to address the detected discrepancy between the first and second composite datasets.

[0306] Furthermore, the control circuitry of one or more aspects of the execution process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) may further enable the display of the current state of the instrument data (e.g., FTF data and / or FTC data) relative to best practice equivalents. Figure 24In the example shown, the current value of FTC—represented by circle 4020—is depicted in real time against a gauge 4021 with an indicator 4022 representing the best practice FTC. Similarly, the current value of FTF—represented by circle 4023—is depicted against a gauge 4024 with an indicator 4025 representing the best practice FTF. This information can be overlaid on the video feed in the surgical field of view in real time.

[0307] Figure 24 The example shown warns the user that the current FTC is higher than the best practice FTC, and the current FTF is also higher than the best practice FTF. If the current values ​​of FTF and / or FTC reach and / or move beyond a predetermined threshold, the control circuitry of one or more aspects of the execution process 4000 may further warn the current user of the surgical instrument 4600 using auditory, visual, and / or tactile warning mechanisms.

[0308] In some cases, the control circuitry of one or more aspects of the execution process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can further provide projected instrument data to the current user of the surgical instrument 4600 based on current investment data. For example, as Figure 24 As shown, the projected FTF 4015 is determined based on the current value of the FTF, and the current FTF 4015 and the previously collected FTF are further displayed on the graph 4013. Alternatively, the projected FTF circle 4026 can be displayed for the gauge 4024, as shown. Figure 24 As shown.

[0309] In each aspect, the previously collected composite datasets and / or best practice FTFs and / or FTCs are determined based on the prior use of the surgical instrument 4600 in the same surgical procedures and / or other surgical procedures performed by the user, other users within the hospital, and / or users in other hospitals. For example, such data can be used by the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) of performing one or more aspects of the process 4000 by importing from the cloud 104.

[0310] In various aspects, control circuitry (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of the execution process 4000 allows feedback measurements of tissue thickness, compression, and stiffness to be visually overlaid on a screen displaying the real-time feed of the surgical instrument 4600 in the surgical field of view as the jaws of the end effector begin to deform the tissue captured therein during the clamping phase. This visual overlay correlates the visualization data representing tissue deformation with changes in clamping force over time. This correlation helps the user confirm the correct cartridge selection, determine the timing of firing initiation, and determine the appropriate firing rate. This correlation can further inform the adaptive clamping algorithm. Adaptive firing rate variations can be indicated by measured forces and changes in tissue motion adjacent to the jaws of the surgical instrument 4600 (e.g., principal strain, tissue slippage, etc.), with gauges or meters conveying the results overlaid on a screen displaying the real-time feed of the end effector in the surgical field of view.

[0311] In addition to the above, the kinematics of the surgical instrument 4600 can further indicate instrument operation relative to another use or user. The kinematics can be determined via an accelerometer, torque sensor, force sensor, motor encoder, or any other suitable sensor, and can generate various force, velocity, and / or acceleration data for the surgical instrument or its components, for correlation with corresponding visualization data.

[0312] In various aspects, if deviations from best practice surgical techniques are detected from visualization data and / or instrument data, the control circuitry of one or more aspects of the execution process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can enable the presentation of alternative surgical techniques. In at least one example, an adaptive display of instrument movement, force, tissue impedance, and results from the given alternative techniques is presented. When visualization data indicates the detection of a blood vessel and applicator in the surgical field of view, the control circuitry of one or more aspects of the execution process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can further ensure the perpendicularity of the blood vessel relative to the applicator. The control circuitry can suggest changes in position, orientation, and / or rolling angle to achieve the desired perpendicularity.

[0313] In various aspects, the control circuitry (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) executes process 4000 by comparing real-time visualization data with preoperative planning simulations. Users can utilize preoperative patient scans to simulate surgical procedures. Preoperative planning simulations allow users to run according to a specific preoperative plan based on training. The control circuitry can be configured to correlate baselines from preoperative scans / simulations with current visualization data. In at least one example, the control circuitry may employ object boundary tracking to establish correlations.

[0314] When a surgical instrument interacts with tissue and deforms its surface geometry, the change in surface geometry with the position of the surgical instrument can be calculated. For a given change in the position of the surgical instrument upon contact with tissue, the corresponding change in tissue geometry may depend on the subsurface structures in the tissue region contacted by the surgical instrument. For example, in thoracic surgery, the change in tissue geometry in regions with airway substructures differs from that in regions with solid substructures. Typically, stiffer substructures produce smaller changes in surface tissue geometry in response to a given change in the position of the surgical instrument. In various aspects, control circuitry (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to calculate a running average of the change in surgical instrument position versus the change in surface geometry for a given patient, thus providing patient-specific differences. Alternatively or additionally, the calculated running average can be compared with a second set of previously collected data. In some cases, a surface reference can be selected when no change in surface geometry is measured with each change in instrument position. In at least one example, the control circuitry may be configured to determine the location of the substructure based on changes in surface geometry detected in response to a given contact between the tissue region and the surgical instrument.

[0315] Furthermore, control circuitry (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain a set instrument-tissue contact throughout the tissue treatment based on the correlation between a set instrument-tissue contact and one or more tissue surface geometry changes associated with the set instrument-tissue contact. For example, the end effector of surgical instrument 4600 can set the desired compression of the instrument-tissue contact to clamp tissue between its jaws. Corresponding changes in tissue surface geometry can be detected by a visualization system. Furthermore, control circuitry (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visual data indicating tissue surface geometry changes associated with the desired compression. The control circuitry can further enable the closure of motor 4603 ( Figure 22The motor settings are automatically adjusted to maintain changes in tissue surface geometry associated with the desired compression. This arrangement requires continuous interaction between the surgical instrument 4600 and the visualization system to maintain changes in tissue surface geometry associated with the desired compression by continuously adjusting the compression of the jaws on the tissue based on visualization data.

[0316] In yet another example, where the surgical instrument 4600 is a robotic tool attached to a robotic arm of a robotic surgical system (e.g., robotic system 110), the robotic surgical system may be configured to automatically adjust one or more components of the robotic surgical system to maintain contact with a predetermined surface of the tissue based on visualization data derived from changes in the tissue surface geometry detected in response to contact with the predetermined surface of the tissue.

[0317] In various examples, visualization data can be combined with measured instrument data to maintain contact between tissues in position or load control, allowing users to manipulate the tissue to apply a predefined load to it as the instrument moves relative to the tissue. Users can specify whether they wish to maintain contact or maintain pressure, and visual tracking of the instrument, as well as the instrument's internal load, can be used to achieve repositioning without changing fixed parameters.

[0318] refer to Figure 25A and Figure 25B The screen 4601 of the visualization system (e.g., visualization systems 100, 160, 500, 2108) displays a real-time video feed of the surgical field of view during the surgical procedure. For example, the end effector 4642 of the surgical instrument 4600 includes jaws that grip tissue near a tumor identified in the surgical field of view via superimposed MRI images. The jaws of the end effector 4642 include an anvil 4643 and a channel for receiving a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other to capture tissue between the anvil 4643 and the staple cartridge. The captured tissue is then sutured via staples 4644 that can be deployed from the staple cartridge during a firing sequence of the surgical instrument 4600. Furthermore, the captured tissue is cut via a cutting member 4645 that advances distally during the firing sequence but slightly lags behind staple deployment.

[0319] like Figure 25AAs is evident, during the firing sequence, the position and / or movement of the captured tissue and certain internal components of the end effector 4642 (such as the staple 4644 and the cutting member 4645) may not be visible in the general view 4640 of the real-time feed on screen 4601. Some end effectors include windows 4641, 4653, which allow a partial view of the cutting member 4645 at the beginning and end of the firing sequence, but not during the firing sequence. Therefore, the user of the surgical instrument 4600 cannot track the progress of the firing sequence on screen 4601.

[0320] Figure 26 This is a logic flowchart of process 4030 according to at least one aspect of the present disclosure, which depicts a control program or logic configuration that synchronizes the motion of a virtual representation of an end effector component with the actual motion of the end effector component. Process 4030 is typically performed during surgical procedures and includes detecting 4031 the motion of an internal component of the end effector during a firing sequence, for example by overlaying 4032 a virtual representation of the internal component onto the end effector, and 4033 synchronizing the motion of the virtual representation on screen 4601 with the detected motion of the internal component.

[0321] One or more aspects of process 4030 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4030 are performed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4030, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4030. Alternatively or additionally, one or more aspects of the process 4030 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The process 4030 is executed by the control circuit 420. Furthermore, the process 4030 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0322] In various examples, control circuitry for one or more aspects of the execution process 4030 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may receive instrument data indicating the movement of an internal component of the end effector 4642 during its firing sequence. For example, a conventional rotary encoder of the firing motor 4602 may be used to track the movement of the internal component. In other examples, the movement of the internal component may be tracked by a tracking system employing an absolute positioning system. A detailed description of an absolute positioning system is described in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FORCONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” the entire contents of which are incorporated herein by reference. In some examples, one or more position sensors may be used to track the movement of the internal component, and these position sensors may include any number of magnetic sensing elements, such as magnetic sensors classified according to whether they measure the total magnetic field or the vector components of the magnetic field.

[0323] In various aspects, process 4030 includes an overlay trigger. In at least one example, the overlay trigger detects tissue being captured by end effector 4642. If tissue captured by end effector 4642 is detected, process 4030 overlays a virtual representation of cutting member 4645 at its initial position onto end effector 4642. Process 4030 further includes projecting pin lines to outline the location where the pin will be deployed into the captured tissue. Furthermore, in response to user activation of the firing sequence, process 4030 causes the virtual representation of cutting member 4645 to move distally, simulating the actual movement of cutting member 4645 within end effector 4642. When the pin is deployed, process 4030 converts unfired pins into fired pins, allowing the user to visually track pin deployment and the advance of cutting member 4645 in real time.

[0324] In various examples, for instance, control circuitry for one or more aspects of the execution process 4030 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may be activated by closing motor 4603 upon instruction. Figure 22Tissue capture by the end effector 4642 is detected by instrument data of the force applied to the jaws of the end effector 4642 by the closing motor drive assembly 4605. The control circuitry can further determine the position of the end effector within the surgical field of view based on visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the end effector position can be determined relative to a reference point in the tissue (e.g., a critical structure).

[0325] In any case, the control circuitry causes a virtual representation of the internal components to be superimposed on the end effector 4642 on screen 4601 at a position commensurate with the position of the internal components within the end effector. The control circuitry further causes the projected virtual representation of the internal components to move synchronously with the internal components during the firing sequence. In at least one example, synchronization is improved by incorporating markers on the end effector 4642, which the control circuitry can use as reference points to determine where to superimpose the virtual representation.

[0326] Figure 25B An enhanced view 4651 shows a real-time feed of the surgical field of view on screen 4601. Figure 25B In the example of the enhanced view 4651, virtual representations of pins 4644 and cutting members 4645 are superimposed on the end effector 4642 during the firing sequence. The superimposition distinguishes fired pins 4644a from unfired pins 4644b and complete cutting lines 4646a from projected cutting lines 4646b that track the progress of the firing sequence. The superimposition further shows the starting point of pin lines 4647 and the projected ends 4649 of pin lines that do not reach the tissue end. Furthermore, based on the superimposition of the tumor MRI image, a safe margin distance “d” between the tumor and the projected cutting lines 4646b is measured and presented along with the superimposition. The superimposed safe margin distance “d” assures the user that all tumor tissue will be removed.

[0327] like Figure 25B As shown, the control circuitry is configured to enable the visualization system to continuously reposition the virtual representation of internal components in relation to the actual movement of those components. Figure 25B In the example, the overlay shows the completed cutting line 4646 slightly behind the fired pin line 4644a by a distance "d1", which assures the user that the firing sequence is proceeding correctly.

[0328] Now for reference Figure 27A visualization system (e.g., visualization systems 100, 160, 500, 2108) may employ tool illumination 4058 and camera 4059 to detect and / or define the cannula position. Based on the determined cannula position, the user can be guided to the most appropriate cannula port to accomplish the intended function based on time efficiency, location of critical structures, and / or avoidance or risk.

[0329] Figure 27 Three cannula positions (cannula 1, cannula 2, and cannula 3) are shown, extending through the body wall 4050 at different positions and orientations relative to the body wall and relative to the critical structure 4051 in the cavity 4052 within the body wall 4050. Cannulas are indicated by arrows 4054, 4055, and 4056. An illumination tool 4058 can be used to detect the cannula position by using cascaded light or images of the surrounding environment. Additionally, the light source of the illumination tool 4058 can be a rotatable light source. In at least one example, the light source of the illumination tool 4058 and a camera 4059 are used to detect the distance of the cannula relative to a target position (e.g., critical structure 4051). In various examples, if a more preferred instrument position is determined based on visualization data obtained from the recording of light projected by the light source of the illumination tool 4058 by the camera 4059, the visualization system can suggest changes in instrument position. Screen 4060 can display the distance between the cannula and the target tissue, whether the tool entry through the cannula is acceptable, the risks associated with using the cannula, and / or the expected operation time for using the cannula. This can help the user select the best cannula for introducing surgical tools into lumen 4052.

[0330] In various aspects, the surgical hub (e.g., surgical hubs 2106, 2122) may suggest the optimal cannula for inserting a surgical instrument into lumen 4052 based on user characteristics, such as those received from a user database. User characteristics include user hand dominance, patient-side user preferences, and / or user body characteristics (e.g., height, arm length, range of motion). The surgical hub may utilize these characteristics, location and orientation data, and / or location data of key structures of available cannulas to select the optimal cannula for inserting the surgical instrument in an effort to reduce user fatigue and improve efficiency. In various aspects, if the user inverts the end effector orientation, the surgical hub may further invert the control of the surgical instrument.

[0331] Surgical hubs can reconfigure the output of surgical instruments based on visualization data. For example, if visualization data indicates that a surgical instrument is retracting or being used to perform a different task, the surgical hub can disable the therapeutic energy output of the surgical instrument.

[0332] In various aspects, visualization systems can be configured to track blood surfaces or estimate blood volume based on reflected IR or red light wavelengths to depict blood according to measurements of non-blood surfaces and surface geometry. This can be reported as an absolute static measurement or rate of change to provide quantitative data on the amount and extent of bleeding.

[0333] refer to Figure 28 Various elements of the visualization system (e.g., visualization systems 100, 160, 500, 2108) (e.g., structured light projector 706 and camera 720) can be used to generate visualization data of anatomical organs, thereby generating a virtual 3D structure 4130 of the anatomical organs.

[0334] As described herein, structured light in the form of stripes or lines can be projected from a light source and / or projector 706 onto the surface 705 of the target anatomical structure to identify the shape and contour of the surface 705. This can be analogous in various respects to imaging device 120. Figure 1 The camera 720 can be configured, for example, to detect the pattern of light projected onto the surface 705. The way the projected pattern deforms upon impact with the surface 705 allows the vision system to calculate depth and surface information of the target's anatomy.

[0335] Figure 29 A logic flowchart of process 4100, which describes a control procedure or logic configuration according to at least one aspect of this disclosure, is provided. In various cases, process 4100 identifies 4101 a surgical procedure and identifies 4102 the anatomical organ targeted by the surgical procedure. Process 4100 further generates 4104 a virtual 3D structure 4130 of at least a portion of the anatomical organ, identifies 4105 the anatomical structure of at least a portion of the anatomical organ associated with the surgical procedure, couples the anatomical structure 4106 to the virtual 3D structure 4130, and overlays 4107 a surgical layout plan determined based on the anatomical structure onto the virtual 3D structure 4130.

[0336] One or more aspects of process 4100 may be executed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4100 are executed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4100, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4100. Alternatively or additionally, one or more aspects of the process 4100 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2CThe control circuit 420) executes the process. Furthermore, one or more aspects of the process 4100 may be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0337] In all respects, process 4100 can be implemented via a computer-based interactive surgical system 2100. Figure 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4100 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0338] Control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) for one or more aspects of the execution process 4100 can identify 4101 the surgical procedure and / or 4102 the anatomical organ targeted by the surgical procedure by retrieving such information from a database of stored information or by obtaining information directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., it may include a cloud 2104 that may include a remote server 2113 coupled to storage device 2105). In at least one example, the database includes a hospital EMR.

[0339] In one aspect, surgical system 2200 includes a surgical hub 2236 connected to multiple operating room devices, such as visualization systems located in the operating room (e.g., visualization systems 100, 160, 500, 2108). In at least one example, surgical hub 2236 includes a communication interface for communicatively coupling surgical hub 2236 to the visualization system, cloud 2204, and / or remote server 2213. Control circuitry of surgical hub 2236 performing one or more aspects of process 4100 can identify 4101 the surgical procedure and / or identify 4102 the anatomical organ targeted by the surgical procedure by retrieving such information from a database stored in cloud 2204 and / or remote server 2213.

[0340] Control circuitry in one or more aspects of the execution process 4100 may cause a visualization system (e.g., visualization systems 100, 160, 500, 2108) to perform an initial scan of at least a portion of the anatomical organ to generate a three-dimensional (“3D”) construct 4130 of at least a portion of the anatomical organ targeted by the surgical procedure 4104. Figure 28 In the example shown, the anatomical organ is the stomach 4110. Control circuitry enables one or more elements of the visualization system (e.g., a structured light projector 706 and a camera 720 utilizing structured light 4111) to generate visualization data by performing a scan of at least a portion of the anatomical organ when the camera is introduced into the body. A 3D reconstruction of at least a portion of the anatomical organ can be generated using current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), visualization data from previous similar surgeries performed on the same or other patients, and / or user input.

[0341] Furthermore, control circuitry in one or more aspects of the execution process 4100 identifies 4105 anatomical structures of at least a portion of the anatomical organ relevant to the surgical procedure. In at least one example, the user may use any suitable input device to select the anatomical structure. Alternatively or additionally, the visualization system may include one or more imaging devices 120 having spectral cameras (e.g., hyperspectral, multispectral, or selective spectral cameras) configured to detect reflected spectral waveforms and generate images based on molecular responses to different wavelengths. Control circuitry may utilize the light absorption or refraction properties of tissues to distinguish different tissue types of the anatomical organ, thereby identifying relevant anatomical structures. Furthermore, control circuitry may utilize current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from previous similar surgeries performed on the same or other patients, and / or user input to identify relevant anatomical structures.

[0342] The identified anatomical structures can be those within the surgical field of view and / or selected by the user. In various examples, the location tracking of the relevant anatomical structures can extend beyond the current visible view of the camera pointing towards the surgical field of view. In one example, this is achieved either by using common visible-coupled landmarks or by using secondary-coupled motion tracking. For example, secondary tracking can be accomplished via a secondary imaging source, calculation of range motion, and / or via pre-established beacons measured by a second visualization system.

[0343] As mentioned above Figure 14In more detail, the visualization system can utilize a structured light projector 706 to project an array of patterns or lines, wherein a camera 720 can determine the distance to the target location. The visualization system can then emit a pattern or line of known size at a set distance equal to the determined distance. Furthermore, a spectral camera can determine the size of the pattern, which can vary according to the light absorption or refraction characteristics of the tissue at the target location. The difference between the known size and the determined size indicates the tissue density at the target location, which indicates the tissue type at the target location. Control circuitry of one or more aspects of the execution process 4100 can identify relevant anatomical structures based at least in part on the tissue density determined at the target location.

[0344] In at least one example, the detected abnormal tissue density may be associated with a disease state. Furthermore, the control circuitry selects, updates, or modifies one or more settings of a surgical instrument used to treat the tissue based on the tissue density detected via visualization data. For example, the control circuitry may alter various clamping and / or firing parameters of a surgical stapler used for suturing and cutting tissue. In at least one example, the control circuitry may slow down the firing sequence and / or allow more clamping time based on the tissue density detected by visualization data. In various examples, the control circuitry may alert the user of the surgical instrument to abnormal tissue density by, for example, displaying on the screen instructions to reduce the bite size, increase or decrease the energy delivery output of the electrosurgical instrument, and adjust the jaw closure amount. In another example, if the visualization data indicates that the tissue is adipose tissue, the instruction could be to increase power to reduce the energy application time.

[0345] Furthermore, the identification of the surgical procedure type facilitates the control circuitry's identification of the target organ. For example, if the surgery is a left upper lobectomy, the lung is likely the target organ. Therefore, the control circuitry will only consider visual and non-visual data related to the lung and / or the tools typically used in such surgeries. Additionally, knowledge of the surgical procedure type can better inform other image fusion algorithms regarding tumor location and suture placement.

[0346] In various aspects, knowledge of the operating table position and / or airflow pressure can be used by control circuitry in one or more aspects of the execution process 4100 to establish baseline positions of target anatomical organs and / or related anatomical structures identified based on visualization data. Movement of the operating table (e.g., moving the patient from a flat position to the reverse Trend-Lombard position) can cause deformation of anatomical structures, which can be tracked and compared to the baseline to continuously inform the position and status of target organs and / or related anatomical structures. Similarly, changes in intracavitary airflow pressure can interfere with baseline visualization data of target organs and / or related anatomical structures within the body cavity.

[0347] The control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) may perform one or more aspects of a process that derives baseline visualization data of the patient’s target organs and / or related anatomical structures on the operating table during surgical procedures, determines changes in the operating table position, and re-derives baseline visualization data of the patient’s target organs and / or related anatomical structures in the new position.

[0348] Similarly, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may perform one or more aspects of a process that, during surgical procedures, derives baseline visualization data of the patient’s target organs and / or relevant anatomical structures on the operating table, determines changes in the insufflation pressure within the patient’s body cavity, and re-derives baseline visualization data of the patient’s target organs and / or relevant anatomical structures at a new insufflation pressure.

[0349] In various cases, the control circuitry of one or more aspects of the execution process 4100 can couple the identified anatomical structures to the virtual 3D structure by overlaying landmarks or markers onto the virtual 3D structure of the organ to indicate the location of the anatomical structures, such as... Figure 28 As shown. The control circuitry also allows user-defined structures and tissue planes to be superimposed on the virtual 3D construct. In various aspects, hierarchical structures of tissue types can be established to organize anatomical structures identified on the virtual 3D construct. Table 1, provided below, lists exemplary hierarchical structures for the lungs and stomach.

[0350]

[0351] In various aspects, the relevant anatomical structures identified on the virtual 3D construct can be renamed and / or repositioned by the user to correct errors or according to preference. In at least one example, the correction can be voice-activated. In at least one example, the correction is recorded for future machine learning.

[0352] In addition to the above, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of the execution process 4100 may overlay 4107 a surgical layout plan (e.g., layout plan 4120) onto a virtual 3D construct of the target organ (e.g., stomach 4110). In at least one example, the virtual 3D construct is displayed on a different screen from the screen displaying the live feed / view of the surgical field of view in the visualization system. In another example, a screen may alternately display the live feed of the surgical field of view and the 3D construct. In such examples, the user may alternate between the two views using any suitable input device.

[0353] exist Figure 28In the example shown, the control circuitry has determined that the surgical procedure is a sleeve gastrectomy and the target organ is the stomach. During the initial abdominal scan, the control circuitry uses visualization data (e.g., structured light data and / or spectral data) to identify the stomach, liver, spleen, greater curvature, and pylorus. This is informed by understanding the surgery and the structures of interest.

[0354] Visualized data (e.g., structured light data and / or spectral data) may be utilized by control circuitry to identify the stomach 4110, liver, and / or spleen by comparing the current structured light data with previously stored structured light data associated with such organs. In at least one example, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may utilize structured light data representing characteristic anatomical contours of the organ and / or spectral data representing tissue features beneath the characteristic surface to identify anatomical structures associated with a surgical layout plan (e.g., layout plan 4120).

[0355] In at least one example, the visualization data can be used to identify the pyloric vein 4112 indicating location 4113 of the pylorus 4131, the gastroepiploic vessel 4114 indicating location 4115 of the greater curvature of the stomach 4110, the bend 4116 in the right gastric vein indicating location 4117 of the notch angle 4132, and / or the location 4119 of the His angle 4121. Control circuitry can assign landmarks to one or more of the identified locations. In at least one example, such as Figure 28 As shown, the control circuitry enables the visualization system to overlay markers at positions 4113, 4117, and 4119 on the virtual 3D structure of the stomach 4110 generated using visualization data, as described above. In various aspects, the markers can be synchronously overlaid on the virtual 3D structure and the surgical field of view, allowing the user to switch between views without losing sight of the markers. The user can zoom out in the view displayed on the screen showing the virtual 3D structure to display the overall layout plan, or zoom in to display a portion similar to the surgical field of view. The control circuitry continuously tracks and updates the markers.

[0356] In sleeve gastrectomy, the surgeon typically sutures the gastric tissue 4 cm or approximately 4 cm from the pylorus. Prior to suturing, at the start of the sleeve gastrectomy, an energy device is introduced into the patient's abdominal cavity to dissect the gastroepiploic artery and omentum, away from the greater curvature and 4 cm or approximately 4 cm from the pylorus. As described above, the control circuitry, having identified location 4113, automatically superimposes the end effector of the energy device at 4 cm or approximately 4 cm from location 4113. The superposition of the end effector of the energy device, or any suitable marker, at 4 cm or approximately 4 cm from the pylorus identifies the starting position of the sleeve gastrectomy.

[0357] As the surgeon dissects along the greater curvature of the stomach, the control circuitry causes the superimposed end effector of the landmark and / or energy device at position 4113 to be removed. As the surgeon approaches the spleen, a distance indicator is automatically superimposed on the virtual 3D anatomical view and / or surgical field of view. The control circuitry allows the distance indicator to identify distances of 2 cm from the spleen. For example, when the anatomical path reaches or is about to reach a distance of 2 cm from the spleen, the control circuitry can cause the distance indicator to flash and / or change color. The distance indicator superimposed remains until the user reaches position 4119 at angle 4121.

[0358] refer to Figure 30 Once the surgical suture device is introduced into the abdominal cavity, the control circuitry can utilize visualization data to identify the pylorus 4131, the notch angle 4132, the greater curvature 4133 of the stomach 4110, the lesser curvature 4134 of the stomach 4110, and / or other anatomical structures associated with sleeve gastrectomy. Overlays of the probes may also be displayed. The introduction of surgical instruments into the body cavity (e.g., the introduction of a surgical suture device into the abdominal cavity) can be detected by the control circuitry based on visualization data of visual cues (such as unique colors, markings, and / or shapes) on the end effector. The control circuitry can identify the surgical instruments in a database storing such visual cues and corresponding visual cues. Alternatively, the control circuitry can prompt the user to identify the surgical instruments inserted into the body cavity. Alternatively, the surgical cannula facilitating access into the body cavity may include one or more sensors for detecting surgical instruments inserted through it. In at least one example, the sensors include an RFID reader configured to identify the surgical instruments from an RFID chip on the surgical instruments.

[0359] In addition to identifying landmarks of relevant anatomical structures, the visualization system can overlay a surgical layout plan 4135 onto a 3D view of key structures and / or the surgical field of view, which can be in the form of a proposed treatment path. Figure 30 In the example, the surgical procedure is a sleeve gastrectomy, and the surgical layout plan 4135 is in the form of three resection paths 4136, 4137, 4138 and the corresponding resulting capacity of the sleeve gastrectomy.

[0360] like Figure 30 As shown, the distances (a, a1, a2) from the pylorus 4131 to the starting points used to form the sleeve stomach are represented. Each starting point produces a different sleeve stomach size (e.g., for starting points 4146, 4147, 4148, at distances a, a1, a2 from the pylorus 4131, they are 400cc, 425cc, and 450cc, respectively). In one example, the control circuit prompts the user to input a size selection, and in response, presents a surgical layout plan, which may be in the form of a resection path, producing the selected sleeve stomach size. In another example, as... Figure 30As shown, the control circuit presents multiple resection paths 4136, 4137, 4138 and corresponding sleeve gastric dimensions. The user can then select one of the given resection paths 4136, 4137, 4138, and in response, the control circuit removes the unselected resection path.

[0361] In yet another example, the control circuitry allows the user to adjust a given resection path on a screen that displays the resection path superimposed on a virtual 3D construct and / or surgical field of view. The control circuitry can then calculate the sleeve stomach size based on these adjustments. Alternatively, in yet another example, the user is allowed to select a starting point to form the sleeve stomach at a desired distance from the pylorus 4131. In response, the control circuitry calculates the sleeve stomach size based on the selected starting point.

[0362] For example, a resection path can be presented by having the visualization system overlay the resection path onto a virtual 3D structural view and / or a surgical field of view. Conversely, a given resection path can be removed by having the visualization system remove the overlay of such a resection path from the virtual 3D structural view and / or the surgical field of view.

[0363] Still referencing Figure 30 In some examples, once the end effector of the surgical stapler grips the gastric tissue selected between the given starting points 4146, 4147, 4147 and the end position 4140 at a predefined distance from the notch angle 4132, the control circuit presents information about gripping and / or firing the surgical stapler. In at least one example, such as Figure 24 As shown, a composite dataset 4012 from visualization data 4010 and instrument data 4011 can be displayed. Alternatively, FTC values ​​and / or FTF values ​​can be displayed. For example, the current value of FTC—represented by circle 4020—can be plotted in real time for a gauge 4021 with an indicator 4022 representing best practice FTC. Similarly, the current value of FTF—represented by circle 4023—can be plotted for a gauge 4024 with an indicator 4025 representing best practice FTF.

[0364] After the surgical stapler is fired, suggestions for selecting a new cartridge can be presented on the surgical stapler's screen or any screen of the visualization system, as described in more detail below. When the surgical stapler is removed from the abdominal cavity, the selected cartridge is reloaded, and reintroduced into the abdominal cavity, a distance indicator—identifying a constant distance (d) from multiple points along the lesser curvature 4134 of the stomach 4110 to the selected resection path—is overlaid on the virtual 3D construct view and / or surgical field of view. To ensure the correct orientation of the surgical stapler's end effector, the distance from the target to the distal end of the surgical stapler's end effector and the distance from the proximal end to the previously fired staple line are overlaid on the virtual 3D construct view and / or surgical field of view. This process is repeated until the resection is complete.

[0365] One or more distances, given and / or calculated by control circuitry, can be determined based on stored data. In at least one example, the stored data includes preoperative data, user preference data, and / or data from previous surgical procedures performed by the user or other users.

[0366] refer to Figure 31 According to at least one aspect of this disclosure, process 4150 describes a control procedure or logical configuration for determining a resection path for removing a portion of an anatomical organ. Process 4150 identifies 4151 the anatomical organ targeted by the surgery, identifies 4152 the anatomical structure of the anatomical organ associated with the surgery, and determines 4153 a surgical resection path for removing a portion of the anatomical organ by surgical instruments, as described in more detail elsewhere herein in conjunction with process 4100. The surgical resection path is determined based on the anatomical structure. In at least one example, the surgical resection path includes different starting points.

[0367] One or more aspects of process 4150 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4150 are performed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4150, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4150. Alternatively or additionally, one or more aspects of the process 4150 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2CThe process 4150 is executed by the control circuit 420. Furthermore, the process 4150 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0368] refer to Figures 32A to 32D The control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of execution process 4100 or process 4150 can utilize dynamic visualization data to update or modify the surgical layout plan in real time during implementation. In at least one example, the control circuitry, based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization systems 100, 160, 500, 2108), sets a pre-defined resection path for removing a portion of an organ or an abnormality (e.g., a tumor or region) using surgical instruments. Figure 32B ) Modified to an alternative resection path ( Figure 32D This visualization system tracks the progress of the removed tissue and surrounding tissue. Modification of the resection path can be triggered by the movement of a critical structure (e.g., a blood vessel) into the resection path. For example, tissue resection procedures sometimes result in tissue inflammation that alters the tissue's shape and / or volume, leading to displacement of critical structures (e.g., blood vessels). Dynamic visualization data enables control circuitry to detect changes in the position and / or volume of critical structures and / or related anatomical structures near the established resection path. If changes in position and / or volume cause a critical structure to move into the resection path or within the safe margins of the resection path, the control circuitry modifies the established resection path by selecting or at least suggesting an alternative resection path for surgical instruments.

[0369] Figure 32A A live view 4201 of the surgical field of view on screen 4230 of the visualization system is shown. Surgical instruments 4200 are introduced into the surgical field of view to remove the target region 4203. An initial planned layout 4209 for removing this region is overlaid on the live view 4201, as shown. Figure 32B An enlarged view of region 4203 is shown. Region 4203 is surrounded by key structures 4205, 4206, 4207, and 4208. (See attached image.) Figure 32B As shown, the initial planned layout 4209 extends a cut-off path around region 4203 with a predefined safety edge. The cut-off path avoids crossing or penetrating critical structures by extending on the outside (e.g., critical structure 4208) or inside (e.g., critical structure 4206) of the critical structure. As described above, the initial planned layout 4209 is determined by control circuitry based on visualization data from a visualization system.

[0370] Figure 32CA real-time view 4201' of the surgical field of view on screen 4230 of the visualization system is shown at a later time (00:43). The end effector 4202 of the surgical instrument 4200 removes tissue along a predefined resection path defined by layout plan 4209. For example, volume changes in tissue including region 4203 due to tissue inflammation cause critical structures 4206 and 4208 to shift into the predefined resection path. In response, the control circuitry provides an alternative resection path 4210 bypassing the critical structures 4206 and 4208, which protects the critical structures 4206 and 4208 from damage, such as... Figure 32D As shown. In various examples, alternative resection paths can be provided to optimize the amount of healthy tissue to be preserved, and users can be guided to ensure they do not touch critical structures. This will minimize bleeding and thus reduce surgical time and stress in dealing with unforeseen circumstances, while balancing the impact on remaining organ capacity.

[0371] In various aspects, control circuitry executing one or more aspects of one or more processes described herein may receive and / or derive visualization data from multiple imaging devices of the visualization system. The visualization data facilitates the tracking of critical structures outside the real-time view of the surgical field of view. Common landmarks allow the control circuitry to incorporate visualization data from multiple imaging devices of the visualization system. In at least one example, secondary tracking of critical structures outside the real-time view of the surgical field of view may be achieved, for example, through secondary imaging sources, calculation of range motion, or through pre-established beacons / landmarks measured by a second system.

[0372] Normal reference Figures 33 to 35 According to at least one aspect of this disclosure, a logic flowchart of process 4300 depicts a control procedure or logic configuration for presenting or overlaying parameters of surgical instruments onto or near a given surgical resection path. Process 4300 is typically performed during surgery and includes 4301 identifying the anatomical organ targeted by the surgery, 4302 identifying anatomical structures relevant to the surgery based on visualization data from at least one imaging device, and 4303 providing a surgical resection path for removing a portion of the anatomical organ using surgical instruments. In at least one example, the surgical resection path is determined based on the anatomical structure. Process 4300 further includes 4304 presenting parameters of the surgical instruments according to the surgical resection path. Additionally or alternatively, process 4300 further includes 4305 adjusting the parameters of the surgical instruments according to the surgical resection path.

[0373] One or more aspects of process 4300 may be executed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4300 are executed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4030, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4030. Alternatively or additionally, one or more aspects of the process 4300 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The process 4300 is executed by the control circuit 420. Furthermore, the process 4300 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0374] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of execution process 4300 may identify 4301 the anatomical organ targeted by the surgery, identify 4302 the anatomical structures associated with the surgery based on visualization data from at least one imaging device of a visualization system (e.g., visualization systems 100, 160, 500, 2108), and / or provide 4303 a surgical resection path for removing a portion of the anatomical organ by means of a surgical instrument (e.g., surgical instrument 4600), as elsewhere herein in conjunction with process 4150 ( Figure 31 ), 4100 Figure 29 As described above. Furthermore, the control circuitry of one or more aspects of the execution process 4300 may provide or suggest one or more parameters of the surgical instruments based on the given surgical resection path 4303. In at least one example, the control circuitry presents suggested parameters 4304 of the surgical instruments by superimposing such parameters onto or near the given surgical path, such as... Figure 34 and Figure 35 As shown.

[0375] Figure 34 A virtual 3D structure 4130 of the stomach of a patient undergoing sleeve gastrectomy using surgical instruments 4600 is shown according to at least one aspect of this disclosure. (As in conjunction with...) Figure 28In more detail, for example, various elements of a visualization system (e.g., visualization systems 100, 160, 500, 2108) (e.g., structured light projector 706 and camera 720) can be used to generate visualization data, thereby forming a virtual 3D structure 4130. Relevant anatomical structures (e.g., pylorus 4131, notch angle 4132, His angle 4121) are identified based on visualization data from one or more imaging devices of the visualization system. In at least one example, landmarks are assigned to locations 4113, 4117, 4119 of such anatomical structures by superimposing landmarks onto the virtual 3D structure 4130.

[0376] Furthermore, based on the identified anatomical structures, a surgical resection path 4312 is provided. In at least one example, the control circuit superimposes the surgical resection path 4312 onto the virtual 3D structure 4130, such as... Figure 34 As shown elsewhere in this document, the given surgical path can be automatically adjusted based on the desired volume output. Furthermore, the projection edges can be automatically adjusted based on key structures and / or tissue abnormalities automatically identified by control circuitry from visualization data.

[0377] In each aspect, the control circuitry of at least one aspect of the execution process 4300 presents parameters 4314 of the surgical instrument selected according to the surgical resection path 4312 given 4303. Figure 34 In the example shown, parameter 4314 instructs automatic selection of the staple cartridge to be used with surgical instrument 4600 when performing a sleeve gastrectomy based on the given surgical resection path 4303. In at least one example, parameter 4314 includes at least one of staple cartridge size, staple cartridge color, staple cartridge type, and staple cartridge length. In at least one example, the control circuitry presents parameter 4314 of surgical instrument 4304 4304 by superimposing such parameters onto or near the given surgical resection path 4312 4303, as... Figure 34 and Figure 35 As shown.

[0378] In each aspect, the control circuitry of at least one aspect of the execution process 4300 presents tissue parameters 4315 along one or more portions of the surgical resection path 4312. Figure 34 In the example shown, tissue parameter 4315 is presented as tissue thickness by displaying a cross-section taken along line AA, representing the tissue thickness along at least a portion of the surgical resection path 4312. In various aspects, the staple cartridge utilized by surgical instrument 4600 can be selected based on tissue parameter 4315. For example, as... Figure 34As shown, black staple cartridges with larger staple sizes are selected for use in thicker antral muscular tissue, while green staple cartridges with smaller staple sizes are selected for use in the myocardial tissue of the gastric body and fundus.

[0379] Tissue parameters 4315 include at least one of the tissue thickness, tissue type, and volume results of the sleeve stomach generated by the given surgical resection path 4312. Tissue parameters 4315 may be derived from previously captured CT, ultrasound, and / or MRI images of the patient organ and / or from previously known average tissue thickness. In at least one example, surgical instrument 4600 is a smart instrument (similar to smart instrument 2112), and tissue thickness and / or selected cartridge information are transmitted to surgical instrument 4600 for optimizing closure settings, firing settings, and / or any other suitable surgical instrument settings. In one example, tissue thickness and / or selected cartridge information may be transmitted from a surgical hub (e.g., surgical hub 2106, 2122) to surgical instrument 4600, which communicates with a visualization system (e.g., visualization systems 100, 160, 500, 2108) and surgical instrument 4600, such as in combination. Figures 17 to 19 As mentioned above.

[0380] In various examples, the control circuitry of at least one aspect of the execution process 4300 provides an arrangement 4317 of two or more staple cartridge sizes (e.g., 45 mm and 60 mm) based on the tissue thickness determined along at least a portion of the surgical resection path 4312. Furthermore, as... Figure 35 As shown, the control circuit can further present an arrangement 4317 along the given surgical resection path 4312. Alternatively, the control circuit can present a suitable arrangement 4317 along a user-selected surgical resection path. As described above, the control circuit can determine the tissue thickness along the user-selected resection path and provide a staple cartridge arrangement based on the tissue thickness.

[0381] In various aspects, the control circuitry of one or more aspects of the execution process 4300 can provide a surgical resection path, or optimize the selected surgical resection path, to minimize the number of staples in the staple cartridge arrangement 4317 without compromising the size of the resulting sleeve stomach beyond a predetermined threshold. Reducing the number of staples used decreases surgical time and cost, and reduces patient trauma.

[0382] Still referencing Figure 35Arrangement 4317 includes a first staple cartridge 4352 and a last staple cartridge 4353 defining the start and end points of surgical resection path 4312. If only a small portion of the last staple cartridge 4353 of the given staple cartridge arrangement 4317 is needed, control circuitry can adjust the surgical resection path 4312 to eliminate the need for the last staple cartridge 4353 without compromising the size of the resulting sleeve stomach beyond a predetermined threshold.

[0383] In various examples, the control circuitry of at least one aspect of the execution process 4300 presents a virtual firing of the given staple cartridge arrangement 4317, which virtually divides the virtual 3D structure 4130 into a retention portion 4318 and a removal portion 4319, such as Figure 35 As shown. The retained portion 4318 is a virtual representation of a sleeve stomach generated by implementing the given surgical resection path 4312 through the firing of the staple cartridge arrangement 4317. Control circuitry may further determine the estimated capacity of the retained portion 4318 and / or the removed portion 4319. The capacity of the retained portion 4318 represents the capacity of the resulting sleeve stomach. In at least one example, the capacity of the retained portion 4318 and / or the removed portion 4319 is derived from visualization data. In another example, the capacity of the retained portion 4318 and / or the removed portion 4319 is determined by a database storing the retained capacity, the removed portion capacity, and the corresponding surgical resection path. The database may be constructed from previous surgeries performed on organs of the same or at least similar size, which have been resected using the same or at least similar resection paths.

[0384] In various examples, a combination of pre-determined average tissue thickness data based on organ situational awareness, as described in more detail above, combined with volume analysis from visualization sources and secondary imaging from CT, MRI, and / or ultrasound, can be used to select the first staple cartridge for placement 4317, if available to the patient. In addition to visualization data, firing of subsequent staple cartridges in placement 4317 can be optimized using instrument data from previous firings. For example, instrument data that can be used to supplement volume measurements includes FTF, FTC, current consumption of the motor driving firing and / or closure, end effector closure gap, firing rate, tissue impedance measurements on the jaws, and / or waiting or pause times during surgical instrument use.

[0385] In various examples, visualization data (e.g., structured light data) can be used to track changes in surface geometry within tissue treated by a surgical instrument (e.g., surgical instrument 4600). Additionally, visualization data (e.g., spectral data) can be used to track key structures beneath the tissue surface. Structured and / or spectral data can be used to maintain established instrument-tissue contact throughout the tissue treatment.

[0386] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between its jaws. Once the desired tissue-instrument contact is confirmed by user input, for example, visualization data of the end effector and surrounding tissue associated with the desired tissue-instrument contact can be used to automatically maintain the desired tissue-surface contact in at least a portion of the tissue treatment. The desired tissue-surface contact can be automatically maintained, for example, by slight manipulation of the position, orientation, and / or FTC parameters of the end effector 4642.

[0387] When the surgical instrument 4600 is a handheld surgical instrument, it can be displayed, for example, on the display 4625 of the surgical instrument 4600. Figure 22 The surgical instrument 4600 provides positional and / or orientation manipulation to the user in the form of instructions. The surgical instrument 4600 may also issue an alarm when user manipulation is required to re-establish desired tissue-surface contact. Simultaneously, non-user manipulation (e.g., manipulation of FTC parameters and / or joint motion angles) may be transmitted from the surgical hub 2106 or visualization system 2108 to the controller 4620 of the surgical instrument 4600. The controller 4620 then enables the motor driver 4626 to perform the desired manipulation. In cases where the surgical instrument 4600 is a surgical tool coupled to a robotic arm of the robotic system 2110, positional and / or orientation manipulation may be transmitted from the surgical hub 2106 or visualization system 2108 to the robotic system 2110.

[0388] Main reference Figures 36A to 36C This illustrates the firing of a surgical instrument 4600 loaded with a first staple cartridge 4652 in a staple cartridge arrangement 4317. In the first stage, as... Figure 36A As shown, a first landmark 4361 and a second landmark 4362 are superimposed on the surgical resection path 4312. Landmarks 4361 and 4362 are spaced apart by a distance (d1) defined by the size (e.g., 45) of the staple cartridge 4652, which represents the length of the staple line 4363 to be deployed by the staple cartridge 4652 onto the surgical resection path 4312. Control circuitry for one or more aspects of the execution process 4300 may employ visual data, as described in more detail elsewhere herein, to superimpose landmarks 4361 and 4362 onto the surgical resection path 4312 and continuously track and update their positions relative to predetermined key structures (e.g., anatomical structures 4364, 4365, 4366, 4367).

[0389] During firing, such Figure 36BAs shown, the staples of staple 4363 are deployed into the tissue, and the cutting member 4645 is advanced to cut the tissue along the surgical resection path 4312 between landmarks 4361 and 4362. In various cases, the advancement of the cutting member 4645 causes stretching and / or displacement of the treated tissue. Tissue stretching and / or displacement exceeding a predetermined threshold indicates that the cutting member 4645 is moving too rapidly through the treated tissue.

[0390] Figure 37 This is a logic flowchart of process 4170, which depicts a control procedure or logic configuration for adjusting the firing rate of a surgical instrument to address tissue stretching / displacement during firing. Process 4170 includes monitoring 4171 tissue stretching / displacement during firing of the surgical instrument, and adjusting 4173 firing parameters if 4172 tissue stretching / displacement is greater than or equal to a predetermined threshold.

[0391] One or more aspects of process 4170 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4170 are performed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4170, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4170. Alternatively or additionally, one or more aspects of the process 4170 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The process 4170 is executed by the control circuit 420. Furthermore, the process 4170 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0392] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of the execution process 4170 uses visualization data from a visualization system (e.g., visualization systems 100, 160, 500, 2108) to monitor tissue stretching / displacement of 4171 during the firing of the surgical instrument 4600. Figure 36BIn the example shown, tissue stretching / displacement (d) of 4171 is monitored by tracking distortion in a structured light grid projected onto the tissue during firing and / or tracking of landmarks 4364, 4365, 4366, and 4367, which represent the positions of adjacent anatomical structures, using visualized data. Alternatively, tissue stretching (d) of 4171 can be monitored by tracking the position of landmark 4362 during firing. Figure 36B In the example, tissue stretching / displacement (d) is the difference between the distance (d1) between markers 4361 and 4362 during firing and the distance (d2) between markers 4361 and 4362 during firing. In any case, if 4172 tissue stretching / displacement (d) is greater than or equal to a predetermined threshold, the control circuit adjusts the firing parameters of the surgical instrument 4600 4173 to reduce tissue stretching / displacement (d). For example, the control circuit can cause the controller 4620 to reduce the speed of the firing motor drive assembly 4604 by, for example, reducing the current consumption of the firing motor 4602, which, for example, reduces the advance speed of the cutting member 4645. Alternatively or additionally, the control circuit can cause the controller 4620 to pause the firing motor 4602 for a predetermined period of time to reduce tissue stretching / displacement (d).

[0393] After firing, as Figure 36C As shown, the jaws of the end effector 4642 loosen, and the sutured tissue shrinks due to the firing pin of the suture 4363. Figure 36C The projected nail line length, defined by distance (d1), and the actual nail line, defined by distance (d3), which is less than distance (d1), are shown. The difference between distances d1 and d2 represents the contraction / displacement distance (d').

[0394] Figure 38 This is a logic flowchart of process 4180, which depicts the control procedure or logic configuration for adjusting the given staple cartridge arrangement along the given surgical resection path. Process 4180 includes, after firing the staple cartridge in the given arrangement, monitoring 4081 the contraction / displacement of the sutured tissue along the given surgical resection path, and adjusting the subsequent staple cartridge position along the given surgical resection path.

[0395] One or more aspects of process 4180 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4180 are performed by control circuits (e.g., Figure 2AThe control circuit 400 executes the process 4180, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4180. Alternatively or additionally, one or more aspects of the process 4180 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The process 4180 is executed by the control circuit 420. Furthermore, the process 4180 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with any of the various suitable systems described in this disclosure.

[0396] In various examples, control circuitry for one or more aspects of the execution process 4180 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) monitors the contraction / displacement of the sutured tissue along the given resection path 4312. Figure 36C In the example shown, suture 4363 is deployed from the staple cartridge of staple cartridge arrangement 4317 into the tissue between landmarks 4361 and 4362. The sutured tissue shrinks / displaces a distance (d') when the jaws of end effector 4642 disengage. Distance (d') is the difference between distance (d1) representing the length of suture 4361 given by arrangement 4317 and pre-firing between landmarks 4361 and 4362, and distance (d3) representing the actual length of suture 4363.

[0397] To avoid gaps between consecutive staples, the control circuit adjusts the subsequent staple cartridge position of the given arrangement 4317 along the given surgical resection path 4312. For example, as Figure 36C As shown, the initially given staple 4368 is removed and replaced by an updated staple 4369 that extends through or covers the gap defined by the distance (d'). In various aspects, tissue contraction / displacement (d') of 4181 is monitored by tracking distortion in a structured light grid projected onto the tissue after the jaws of the end effector 4642 are released using visualization data and / or by tracking landmarks 4364, 4365, 4366, 4367 representing the positions of adjacent anatomical structures. Alternatively, the tissue contraction distance (d') of 4181 can be monitored by tracking the position of landmark 4362.

[0398] In various aspects, it may be desirable to use non-visual data from non-visualization systems to validate visual data derived from surgical visualization systems (e.g., visualization systems 100, 160, 500, 2108), and vice versa. In one example, a non-visualization system may include a ventilator that can be configured to measure non-visual data of a patient's lungs, such as volume, pressure, partial pressure of carbon dioxide (PCO2), partial pressure of oxygen (PO2), etc. Validating visual data with non-visualization data provides clinicians with greater confidence that the visual data derived from the visualization system is accurate. Furthermore, validating visual data with non-visualization data allows clinicians to better identify postoperative complications and determine the overall efficiency of the organ, as will be described in more detail below. Validation can also be helpful in segmentectomies or complex lobectomies without clefts.

[0399] In various situations, clinicians may need to remove a portion of a patient's organ to remove critical structures such as tumors and / or other tissues. In one example, the patient's organ could be the right lung. Clinicians might need to remove a portion of the patient's right lung to remove unhealthy tissue. However, clinicians may not want to remove too much of the patient's lung during surgery to ensure that lung function is not significantly impaired. Lung function can be assessed based on peak lung volume per breath, which represents the peak lung capacity. In determining how much lung can be safely removed, clinicians are limited by a predetermined reduction in peak lung capacity beyond which the lung will lose its viability and require total organ removal.

[0400] In at least one example, the surface area and / or volume of the lung are estimated based on visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). The lung surface area and / or volume can be estimated at peak lung volume or peak lung volume per breath. In at least one example, the lung surface area and / or volume can be estimated at multiple points throughout the inspiratory / expiratory cycle. In at least one aspect, prior to the removal of a portion of the lung, the lung surface area and / or volume determined by the visualization system can be correlated with the lung volume determined by the ventilator using both visualization and non-visualization data. For example, correlation data can be used to establish a mathematical relationship between the lung surface area and / or volume derived from visualization data and the lung volume determined by the ventilator. This relationship can be used to estimate the size of the lung portion that can be removed while maintaining the reduction in peak lung volume to a value less than or equal to a predetermined threshold for maintaining lung viability.

[0401] Figure 39A logic flowchart of a procedure 4750 for surgical resection of an organ portion according to at least one aspect of this disclosure is shown. Procedure 4750 is typically performed during surgical procedures. Procedure 4750 may include determining 4752 the portion of the organ to be resected based on visualization data from a surgical visualization system, wherein the resection of this portion is configured to produce an estimated reduction in organ volume. Procedure 4750 may further include determining 4754 a first value of a non-visualization parameter of the organ before resection of the portion, and determining 4756 a second value of the non-visualization parameter of the organ after resection of the portion. Additionally, in some examples, procedure 4750 may further include confirming 4758 the predetermined reduction in volume based on the first value of the non-visualization parameter and the second value of the non-visualization parameter.

[0402] One or more aspects of process 4750 may be executed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4750 are executed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4750, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4750. Alternatively or additionally, one or more aspects of the process 4750 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2C The control circuit 420) executes the process. Furthermore, one or more aspects of the process 4750 may be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0403] In all respects, process 4750 can be implemented via a computer-based interactive surgical system 2100. Figure 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4750 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0404] Figure 41AA group of patients' lungs 4780 are shown. In one embodiment, a clinician may use imaging device 4782 to project a pattern of light 4785 4784 onto the surface of the patient's right lung 4786, such as stripes, grid lines, and / or dots, to enable the determination of the topography or panorama of the surface of the patient's right lung 4786. The imaging device may be similar in various respects to imaging device 120 ( Figure 1 As described elsewhere herein, the projection light array can be used to determine the shape defined by the surface of the patient's right lung 4786, and / or the intraoperative movement of the patient's right lung 4786. In one embodiment, the imaging device 4782 may be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system such as surgical visualization system 100 may utilize surface mapping logic 136 of the control circuitry 133, as described elsewhere herein, to determine the topography or panorama of the surface of the patient's right lung 4786.

[0405] Clinicians can provide the surgical system (such as surgical system 2100) with the type of surgery to be performed, such as a right upper lobectomy. In addition to providing the surgical procedure to be performed, clinicians can also provide the surgical system with the maximum expected volume of the organ to be removed during the surgery. Based on visualization data obtained from imaging device 4782, the type of surgical procedure to be performed, and the maximum expected volume to be removed, the surgical system can provide a resection path 4788 to remove a portion 4790 of the right lung that satisfies all clinician inputs. Other methods for providing surgical resection paths are described elsewhere in this document. The surgical system can consider any number of additional parameters to provide the resection path 4788.

[0406] In various situations, it may be desirable to ensure that the volume of the removed organ produces the desired volume reduction in the patient's organs. To confirm that the removed volume produces the desired volume reduction, non-visual data from non-visual systems can be utilized. In one embodiment, a ventilator can be used to measure peak lung volume in a patient over time.

[0407] In at least one example, a clinician could use the surgical system 2100 to remove a lung tumor during surgery. (As mentioned above...) Figures 13A to 13E The control circuit can identify tumors based on visualized data and provide a surgical resection path with safe margins around the tumor, as described above. Figures 29 to 38As described, the control circuitry can further estimate lung volume at peak lung volume. A ventilator can be used to measure peak lung volume prior to surgery. Using a predetermined mathematical correlation between visually estimated lung volume and lung volume as detected by the ventilator, the control circuitry is able to estimate the reduction in peak lung volume associated with the removal of a portion of the lung, including a safe margin for the tumor and surrounding tissue. If the estimated reduction in lung volume exceeds a predetermined safe threshold, the control circuitry can alert the clinician and / or suggest alternative surgical resection paths that result in a smaller reduction in lung volume.

[0408] Figure 41C A graph 4800 shows the peak lung volume of a patient measured over time. Peak lung volume can be measured by the ventilator before this portion of the organ is removed (t1). Figure 41C In the example described above, at time t1 prior to the resection of portion 4790, the peak lung volume was measured to be 6 L. In this case, where the surgical procedure to be performed is a right upper lobectomy, the clinician may wish to remove only the volume of the patient's lung that results in a predetermined volume reduction, so that the patient's breathing capacity is not impaired. In one implementation, for example, the clinician may wish to remove a portion that results in a reduction of up to approximately 17% in the patient's peak lung volume. Based on the surgical procedure and the desired volume reduction, the surgical system can provide a surgical resection path 4788 that achieves the removal of the lung portion while maintaining a peak lung volume greater than or equal to 83% of the unresected peak lung volume.

[0409] Utilize Figure 41C The ventilator data shown allows clinicians to monitor peak lung volume over time, for example, before (4802) and after (4804) the removal of portion 4790 of the lung. At time t2, portion 4790 of the lung is removed along the given resection path 4788. As a result, the peak lung volume measured by the ventilator decreases. Clinicians can use the ventilator data (peak lung volume before removal 4802 and peak lung volume after removal 4804) to confirm this and ensure that the removed lung volume results in the desired reduction in lung volume. Figure 41C As shown, after resection, peak lung volume decreased to 5L, representing a decrease of approximately 17%, which is roughly the same as the expected volume reduction. Using ventilator data, clinicians have greater confidence that the actual volume reduction is consistent with the expected volume reduction achieved through the given surgical resection path 4788. In other implementations, in cases where there are discrepancies between non-visualized and visualized data, such as a larger-than-expected decrease in peak lung volume (over-resection) or a smaller-than-expected decrease in peak lung volume (under-resection), clinicians can determine whether appropriate action should be taken.

[0410] Now for reference Figure 41BThe patient's right lung 4792 is shown after the resection of portion 4790. Following the resection of portion 4790, a clinician may inadvertently cause an air leak 4794, resulting in air leakage into the space between lung 4794 and the chest wall, leading to pneumothorax 4796. Due to the air leak 4794, the peak lung volume of the patient with each breath will steadily decrease over time as the right lung 4792 collapses. Dynamic surface area / volume analysis of the lung can be performed using visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) to detect the air leak by visually tracking changes in lung volume. The lung volume and / or surface area can be visually tracked at one or more points during the inspiratory / expiratory cycle to detect volume changes indicative of the air leak 4794. In one embodiment, as described above, a projection light array from imaging device 4782 can be used to monitor the movement of the patient's right lung 4786 over time, such as monitoring a decrease in size. In another implementation, the surgical visualization system may utilize surface mapping logic, such as surface mapping logic 136, to determine the topography or panorama of the surface of the patient's right lung 4786, and monitor changes in the topography or panorama over time.

[0411] In one respect, clinicians can use non-visual systems such as ventilators to confirm volume reductions detected by visual systems. (See also...) Figure 41C As described above, peak lung volume can be measured before (4802) and after partial lung resection to confirm that the expected volume reduction is consistent with the actual reduction in lung volume. In the example described above, in the case of an unintentional air leak, peak lung volume can steadily decrease over time (4806). In one case, at time t2 immediately after the resection, the clinician can record a decrease in peak lung volume from 6L to 5L, which is roughly consistent with the expected reduction in lung volume. After the resection, a surgical visualization system can monitor the patient's lung volume over time. If the surgical visualization system determines that a volume change exists, the clinician can, for example, measure peak lung volume again at time t3. At time t3, the clinician can record a decrease in peak lung volume from 5L to 4L, which confirms the data determined from the visualization system, indicating a possible air leak in the right lung (4792).

[0412] Furthermore, the control circuitry can be configured to measure organ efficiency based on both visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing visualized data with the difference in non-visualized data before and after the removal of the portion. In one example, the visualization system can generate a resection path that reduces peak lung volume by 17%. The ventilator can be configured to measure peak lung volume before and after the removal of the portion. Figure 41CIn the scenario shown, there is a decrease of approximately 17% of the peak lung volume (6L to 5L). Since the actual lung volume decrease (17%) is close to a 1:1 ratio with the expected lung volume decrease (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can generate a resection path to reduce the peak lung volume by 17%. However, as an example, the ventilator can measure a decrease in peak lung volume greater than 17% (such as 25%). In this case, the clinician can determine that the lung is not functionally effective because removing this portion of the lung results in a greater decrease in peak lung volume than expected.

[0413] Figure 40 A logical flowchart of process 4760 according to at least one aspect of this disclosure is shown, which is used to estimate the amount of organ volume reduction resulting from the removal of a selected portion of an organ. Process 4760 is similar to process 4750 in many respects. However, unlike process 4750, process 4760 relies on a clinician selecting or providing a surgical resection path for removing a portion of the organ during surgery. Process 4760 includes receiving 4762 input from a user indicating the portion of the organ to be removed. Process 4760 further includes estimating 4764 the amount of organ volume reduction resulting from the removal of that portion. In at least one example, the organ is a patient's lung, and the estimated amount of volume reduction 4762 is the reduction in peak lung volume per breath of the patient's lung. Visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108) may be used to estimate the amount of volume reduction corresponding to the removal of that portion. Process 4760 may further include determining a first value of a non-visual parameter of organ 4766 before the portion is removed, and determining a second value of a non-visual parameter of organ 4768 after the portion is removed. Finally, process 4760 may further include confirming the estimated volume reduction of organ 4768 based on the first value of the non-visual parameter and the second value of the non-visual parameter.

[0414] One or more aspects of process 4760 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4760 are performed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4760, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4760. Alternatively or additionally, one or more aspects of the process 4760 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2CThe control circuit 420) executes the process. Furthermore, one or more aspects of the process 4760 may be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0415] In all respects, process 4760 can be implemented via a computer-based interactive surgical system 2100. Figure 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4760 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0416] In one scenario, a clinician may provide input to a surgical visualization system (such as surgical visualization system 2100) indicating the portion of an organ to be removed. In another scenario, the clinician may draw a resection path on a virtual 3D structure of the organ (such as the virtual 3D structure generated during procedure 4104). In other scenarios, the visualization system may overlay a surgical layout plan, described in more detail elsewhere in this document, which may be in the form of a proposed treatment path. The proposed treatment path may be based on the type of surgery being performed. In one embodiment, the proposed treatment path may provide different starting points and offer different resection paths that the clinician can choose from, similar to resection paths 4146, 4147, and 4148 described elsewhere in this document. The given resection path may be determined by the visualization system to avoid certain critical structures such as arteries. The clinician may select a given resection path until the desired resection path for removing the portion of the organ is completed.

[0417] In one scenario, a surgical visualization system can determine the estimated volume reduction of an organ based on a selected resection path. After removing a predetermined portion along the resection path, clinicians may wish to use non-visual data to confirm that the actual volume reduction corresponds to the estimated volume reduction based on the visualization data. In one implementation, this confirmation can be accomplished using a procedure similar to that described above with respect to the lung (procedure 4750). Peak lung volume is measured before (4802) and after (4804) lung resection, and changes in peak lung volume are compared to determine the actual decrease in peak lung volume. In one example, the surgical visualization system can estimate a 17% reduction in peak lung volume based on a resection path given by the clinician. Before resection, the clinician may record a peak lung volume of 6 L (time t1). After resection, the clinician may record a peak lung volume of 5 L (time t2), representing approximately a 17% decrease in peak lung volume. Using this non-visual / ventilator data, clinicians have greater confidence that the actual volume reduction is consistent with the estimated volume reduction. In other cases where there are discrepancies between non-visualized and visualized data, such as a greater-than-expected decrease in peak lung volume (excessive lung resection) or a smaller-than-expected decrease in peak lung volume (insufficient lung resection), clinicians can determine whether appropriate action should be taken.

[0418] Furthermore, the control circuitry can be configured to measure organ efficiency based on both visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing visualized data with non-visualized data before and after the removal of the portion. In one example, the visualization system can estimate a 17% reduction in peak lung volume based on the clinician's desired resection path. The ventilator can be configured to measure peak lung volume before and after the removal of the portion. Figure 41C In the scenario shown, there is a reduction of approximately 17% in peak lung volume (6L to 5L). Since the actual lung volume reduction (17%) is close to a 1:1 ratio with the estimated lung volume reduction (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can estimate a 17% reduction in peak lung volume based on the clinician's expected resection path. However, as an example, the ventilator can measure a reduction in peak lung volume greater than 17% (such as 25%). In this case, the clinician can determine that the lung is not functionally effective because removing this portion of the lung results in a larger reduction in peak lung volume than expected.

[0419] As described above regarding procedures 4750 and 4760, clinicians can use non-visual data (e.g., measuring peak lung volume before and after partial lung removal using a ventilator) to confirm visual data. Another example of using non-visual data to confirm visual data is through carbon dioxide graphs.

[0420] Figure 42 A graph 4810 shows the change in the partial pressure of carbon dioxide (PCO2) exhaled by the patient over time. In other cases, the partial pressure of oxygen (PO2) exhaled by the patient can be measured over time. Graph 4810 shows PCO2 measured before resection 4812, immediately after resection 4814, and one minute after resection 4816. Figure 42 In the example described above, 4812 minutes prior to the resection, PCO2 was measured at approximately 40 mmHg (at time t1). In this case, where the surgical procedure to be performed was a right upper lobectomy, the visualization system might expect or estimate a 17% reduction in lung volume. The PCO2 level measured by the ventilator could be used to confirm this expected or estimated volume reduction.

[0421] Utilize Figure 42 The ventilator data shown allows clinicians to monitor a patient's PCO2 over time, for example, before (4812) and immediately after (4814) the removal of a portion of the lung (4790). At time t2, part of the lung (4790) has been removed, and therefore, the PCO2 measured by the ventilator may have decreased by 4818. Clinicians can use the ventilator data (PCO2 before removal (4812 at t1) and PCO2 after removal (4814 at t2)) to confirm this and ensure that the actual reduction in lung volume is consistent with the estimated or expected reduction. Figure 42 As shown, immediately following the resection of part 4790 4814, PCO2 decreased 4812, which can be measured as a decrease of approximately 17% of PCO2 (approximately 33.2 mmHg). Using this non-visual / ventilator data, clinicians have greater confidence that the actual reduction in lung volume is consistent with the expected or estimated reduction in lung volume.

[0422] In other cases, clinicians can use non-visualized / PCO2 data to determine differences compared to visualized data. In one case, immediately following the resection of part 4790 at 4814, at time t2, PCO2 can be measured at 4820, which is higher than the PCO2 measured before resection at 4812. The increase in PCO2 may be a result of unintentional bronchial obstruction during surgery, leading to CO2 buildup in the patient's body. In another case, immediately following the resection of part 4790 at 4814, at time t2, PCO2 can be measured at 4822, which is lower than the PCO2 measured before resection at 4812 and lower than expected. The decrease in PCO2 may be a result of unintentional vascular obstruction during surgery, leading to less O2 being delivered to the body and thus less CO2 being produced. In either case, clinicians can take appropriate measures to remedy the situation.

[0423] Changes in PCO2 can also be measured at times other than immediately after resection 4814, such as one minute after resection 4816 (e.g., at time t3). At time t3, other bodily functions (such as the kidneys) compensate for the change in PCO2 due to resection. In this case, PCO2 may be measured at approximately 40 mmHg, or approximately the same as the measurement before resection 4812. The difference between the measured PCO2 at time t3 and the PCO2 before resection 4812 can indicate unintentional obstruction discussed above. For example, at time t3, PCO2 may be measured 4824 as higher than before resection 4812, indicating possible unintentional bronchial obstruction, or PCO2 may be measured 4826 as lower than before resection 4812, indicating possible unintentional vascular obstruction.

[0424] Furthermore, the control circuitry can be configured to measure organ efficiency based on both visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing visualized data with the difference in non-visualized data before and after the resection of the part. In one example, the visualization system could estimate a 17% reduction in lung volume based on the clinician's expected resection path. The ventilator can be configured to measure PCO2 before and after the resection of the part. Figure 42 In the illustrated implementation, PCO2 decreased by approximately 17% immediately following the resection. Since the PCO2 decrease (17%) is close to a 1:1 ratio with the estimated lung volume decrease (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can estimate a 17% reduction in lung volume based on the clinician's expected resection path. However, as an example, the ventilator can measure a PCO2 decrease greater than 17% (such as 25%). In this case, the clinician can determine that the lung is not functionally effective because the resection of that portion of the lung resulted in a greater PCO2 decrease than expected.

[0425] In addition to peak lung volume and PCO2 measurements mentioned above, other non-visual parameters, including blood pressure or EKG data, can be utilized. EKG data will provide approximate frequency data on arterial deformation. This frequency data, showing variations in surface geometry within a similar frequency range, can help identify key vascular structures.

[0426] As described above, it may be desirable to utilize non-visualization data from a non-visualization system to validate visualization data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). In the examples above, non-visualization data provides a means of validating visualization data after a portion of an organ has been removed. In some cases, it may be desirable to supplement visualization data with non-visualization data before removing a portion of an organ. In one example, non-visualization data may be used in conjunction with visualization data to help determine the characteristics of the organ to be operated on. In one aspect, characteristics may be anomalies in organ tissue that may be unsuitable for cutting. Non-visualization data and visualization data can help inform the surgical visualization system and clinicians about areas to avoid when planning the organ resection path. This can be helpful in segmentectomy or complex lobectomy without clefts.

[0427] Figure 43 A logical flowchart of process 4850 according to at least one aspect of this disclosure is shown, which is used to detect tissue abnormalities based on visualized and non-visualized data. Process 4850 is typically performed during surgical procedures. Process 4850 may include receiving 4852 first visualized data from a surgical visualization system in a first state of the organ, and determining 4854 a first value of a non-visualized parameter of the organ in the first state. Process 4850 may further include receiving 4856 second visualized data from the surgical visualization system in a second state of the organ, and determining 4858 a second value of a non-visualized parameter of the organ in the second state. The process may also include detecting 4860 tissue abnormalities based on the first visualized data, the second visualized data, the first value of the non-visualized parameter, and the second value of the non-visualized parameter.

[0428] One or more aspects of process 4850 may be performed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4850 are performed by control circuits (e.g., Figure 2A The control circuit 400 executes the process 4850, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4850. Alternatively or additionally, one or more aspects of the process 4850 may be executed by combinational logic circuitry (e.g., Figure 2B Control circuit 410) and / or sequential logic circuit (e.g., Figure 2CThe control circuit 420) executes the process. Furthermore, one or more aspects of the process 4850 may be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0429] In all respects, the process 4850 can be implemented via a computer-based interactive surgical system 2100. Figure 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4850 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0430] Figure 44A The image shows the right lung 4870 of a patient in a first state 4862. In one example, the first state 4862 may be a contracted state. In another example, the first state 4862 may be a collapsed state. The imaging device 4872 is shown inserted through a cavity 4874 in the patient's chest wall 4876. Clinicians can use the imaging device 4872 to project patterns 4882 4880 of light onto the surface of the right lung 4870, such as stripes, grid lines, and / or dots, to enable the determination of the topography or panorama of the surface of the patient's right lung 4870. The imaging device may be similar in various respects to imaging device 120 ( Figure 1 As described elsewhere herein, the projection light array is used to determine the shape defined by the surface of the patient's right lung 4870, and / or the intraoperative movement of the patient's right lung 4870. In one embodiment, the imaging device 4782 may be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system such as surgical visualization system 100 may utilize surface mapping logic 136 of the control circuitry 133, as described elsewhere herein, to determine the topography or panorama of the surface of the patient's right lung 4786. In a first state 4862 of the right lung 4870, a ventilator may be used to measure parameters of the right lung 4870, such as first state pressure (P1, or positive end-expiratory pressure (PEEP)) or first state volume (V1).

[0431] Figure 44BThe image shows the right lung 4870 of a patient in a second state 4864. In one example, the second state 4864 may be a partially inflated state. In another example, the second state 4864 may be a fully inflated state. The imaging device 4872 may be configured to continuously emit a pattern 4882 of light 4880 onto the surface of the lung 4870, thereby enabling the determination of the topography or panorama of the surface of the patient's right lung 4870 in the second state 4864. In the second state 4864 of the right lung 4870, a ventilator may be used to measure parameters of the right lung 4870, such as a second-state pressure (P2) greater than the first-state pressure P1 and a second-state volume (V2) greater than the first-state volume V1.

[0432] Based on the surface morphology determined from the surgical visualization system and imaging device 4872, and non-visual data determined from the ventilator (pressure / volume), the surgical visualization system can be configured to identify tissue abnormalities in the right lung 4870. In one example, in a first state 4862, the imaging device 4872 can determine the first state morphology 4662 of the right lung 4870 (in... Figure 44A As shown in and Figure 44C (shown in more detail below), and the ventilator can determine the pressure / volume in the first state. In the second state 4864, the imaging device 4872 can determine the morphology of the right lung 4870 in the second state 4864 (in...). Figure 44B As shown in and Figure 44D (As shown in more detail below), and the ventilator can determine a second-state pressure / volume greater than the first-state pressure / volume due to partial or complete lung expansion. Based on the known pressure / volume increase, the visualization system can be configured to monitor morphological changes in the right lung 4870 according to the known pressure / volume increase. In one aspect, this pressure / volume measurement from the ventilator can be correlated with surface deformation of the right lung 4870 to identify disease areas within the lung, thereby helping to inform suture placement.

[0433] In one respect, reference Figure 44B and Figure 44D As pressure increases from P1 to P2 (and volume increases from V1 to V2), the surface topography determined by structured light 4880 has changed compared to the first state 4862. In one example, the light pattern 4882 can be dots, and these dots are spaced apart from each other as the lung size increases. In another example, the light pattern 4882 can be grid lines, and the grid lines are spaced apart or form an outline as the lung size increases. Based on the known increase in pressure, the imaging device can determine the region 4886 that has not changed according to the known increase in pressure and volume. For example, in the imaging device 4872, the pattern 4882 of grid lines and dots is projected onto the surface of the right lung 4870 (e.g., Figures 44A to 44DIn the case shown), the visualization system can be configured to increase the monitoring of the contours of the grid lines and the positioning of the points relative to each other for known pressure / volume. When the visualization system notices irregularities in the spacing of the points or the positioning and curvature of the grid lines, the visualization system can determine that these areas correspond to potential abnormalities in the tissue, such as areas where critical structures 4884 (such as tumors) may be located, or subsurface cavities 4886. In one embodiment, referring to process 4100, where process 4100 identifies 4105 anatomical structures of at least a portion of surgically relevant anatomical organs, process 4100 can identify abnormalities as described above and overlay these abnormalities onto a 3D construct.

[0434] In one example, the patient might have emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs in the lungs (alveoli) are damaged, and over time, the inner walls of the air sacs weaken and rupture, creating larger air spaces instead of many small ones. This reduces the inner surface area of ​​the lungs used for O2 / CO2 exchange, thus reducing the amount of oxygen reaching the bloodstream. Furthermore, the damaged alveoli do not function properly, trapping old air and leaving no room for fresh, oxygen-rich air to enter. The gaps in the lungs of a patient with emphysema represent areas with less tissue thickness, thus affecting the outcome of sutures in that area. The tissue is also weakened, which leads to alveolar rupture and makes it less likely for staples to hold through them.

[0435] As a lung with emphysema expands and contracts, areas with subsurface voids will deform differently with pressure compared to healthy tissue. Using the procedure described above 4850, these weak tissue areas with subsurface voids can be detected, informing clinicians that they should avoid suturing through these areas, reducing the likelihood of postoperative air leakage. The tissue deformability of this procedure 4850 will allow for the detection of these differences, thus guiding the surgeon when placing the suture device.

[0436] In the second example, the patient may have cancer. Prior to surgery, the tumor may have been irradiated, damaging the tissue and surrounding tissue. Irradiation alters the properties of tissue, typically making it harder and less compressible. If the surgeon needs to suture this tissue, the change in tissue stiffness should be taken into account when selecting the type of staple reloading (e.g., harder tissue will require staples with higher shaping).

[0437] When the lungs expand and contract, areas with stiffer tissue will deform differently compared to healthy tissue because the lungs are less compliant in these areas. The tissue deformability of the 4850 procedure will allow for the detection of these differences, thus guiding the surgeon during suture placement and selection of chamber / reload color.

[0438] On the other hand, a memory (such as memory 134) can be configured to store the surface morphology of the lung at known pressures and volumes. In this case, an imaging device such as imaging device 4872 can emit a pattern of light to determine the morphology of the patient's lung surface at a first known pressure or volume. A surgical system such as surgical system 2100 can be configured to compare a first determined morphology at a known first pressure or volume with the morphology stored in memory 134 at a given first pressure or volume. Based on this comparison, a visualization system can be configured to indicate potential abnormalities in the tissue in only a single state. The visualization system can record these potential abnormal regions and continue to determine the morphology of the patient's lung surface at a second known pressure or volume. The visualization system can compare the second determined surface morphology with the morphology stored in memory at a second given pressure or volume and the morphology determined at the first known pressure or volume. If the visualization system determines a potential abnormal region superimposed on the first determined potential abnormal region, the visualization system can be configured to indicate the superimposed region as a potential abnormality with greater confidence based on the comparison at the first and second known pressures or volumes.

[0439] In addition to the above, PO2 measurements from the ventilator can be compared with inflated lung volume (such as V2) and systolic lung volume (such as V1). Volume comparisons can be made using EKG data to compare inspiration and expiration, which can be compared with blood oxygenation. This can also be compared with anesthetic gas exchange measurements to determine the relationship between respiratory volume, oxygen uptake, and sedation. Furthermore, EKG data can provide approximate frequency data on arterial deformation. This frequency data, showing variations in surface geometry within a similar frequency range, can help identify key vascular structures.

[0440] In another implementation, current tracking / surgical information can be compared with preoperative planning simulations. In challenging or high-risk surgeries, clinicians can use preoperative patient scans to simulate surgical approaches. This dataset can be compared with real-time measurements on a display (such as display 146) to help surgeons follow a specific preoperative plan based on training runs. This will require the ability to match baseline boundaries between preoperative scans / simulations and current visualizations. One approach could be simply using object boundary tracking. Insights into how current device-tissue interactions compare to previous interactions (per patient) or anticipated interactions (database or past patients) for tissue type identification, relative tissue deformation assessment, or subsurface structural differences can be stored in memory such as memory 134.

[0441] In one implementation, the surface geometry can be a function of the tool position. A surface reference can be selected when no change in surface geometry is measured for each change in tool position. As the tool interacts with tissue and deforms the surface geometry, the surgical system can calculate the change in surface geometry as a function of tool position. For a given change in tool position upon contact with tissue, the change in tissue geometry may differ in regions with subsurface structures (such as critical structures 4884) than in regions without such structures (such as subsurface voids 4886). In an example such as thoracic surgery, this may be above the airway rather than just in the parenchyma. The surgical system can use a surgical visualization system to calculate a running average of the tool position change versus surface geometry change for a given patient, thus providing a patient-specific difference, or it can compare this value with a second set of previously collected data.

[0442] Exemplary clinical applications

[0443] The various surgical visualization systems disclosed herein can be used in one or more of the following clinical applications. The following clinical applications are non-exhaustive and merely illustrative applications of one or more of the various surgical visualization systems disclosed herein.

[0444] The surgical visualization system disclosed herein can be used in a variety of surgical procedures for various medical specialties, such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, obesity / gastroenterology, and hepatobiliary surgery (HPB). For example, in urological surgeries (such as prostatectomy), the ureter can be detected in fat, or connective tissue and / or nerves can be detected in fat. Similarly, in gynecological oncology surgeries (such as hysterectomy) and colorectal surgeries (such as low anterior resection (LAR)), the ureter can be detected in fat and / or connective tissue. In thoracic surgeries (such as lobectomy), blood vessels can be detected in the lungs or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In obesity surgeries, blood vessels can be detected in fat. For example, in HPB procedures (such as hepatectomy or pancreatectomy), blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0445] In one example, a clinician might want to remove an endometrial fibroid. Based on a preoperative magnetic resonance imaging (MRI) scan, the clinician knows that the fibroid is located on the surface of the intestine. Therefore, the clinician might want to know intraoperatively which tissues constitute part of the intestine and which constitute part of the rectum. In such cases, a surgical visualization system, as disclosed herein, can indicate the different types of tissues (intestine and rectum) and convey this information to the clinician via an imaging system. Furthermore, the imaging system can determine and transmit the proximity of the surgical apparatus to the selected tissue. In such cases, a surgical visualization system can provide increased surgical efficiency without serious complications.

[0446] In another example, clinicians (e.g., gynecologists) may keep away from certain anatomical areas to avoid getting too close to critical structures, and therefore may be unable to remove, for example, all endometriosis. Surgical visualization systems, as disclosed herein, allow gynecologists to reduce the risk of getting too close to critical structures, enabling them to use surgical instruments close enough to remove all endometriosis, which improves patient outcomes (democratizing surgery). Such systems allow surgeons to “keep moving” during surgery rather than repeatedly stopping and starting in order to identify areas to avoid, especially during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and given the presentation and / or thickness of the tissues involved, the system may be particularly useful for hysterectomy and endometriosis removal.

[0447] In another example, clinicians may risk dissecting blood vessels too close to the target lobe, potentially affecting blood supply to lobes other than the target lobe. Furthermore, patient-to-patient anatomical differences may lead to dissections based on the specific patient, affecting blood vessels (e.g., branches) in different lobes. Surgical visualization systems as disclosed herein enable the identification of the correct blood vessels at the desired location, allowing clinicians to perform dissections with appropriate anatomical certainty. For example, the system can confirm the correct blood vessel is in the correct location, and the clinician can then safely separate the vessel.

[0448] In another example, due to the uncertainty of the anatomical structure of blood vessels, clinicians may perform multiple dissections before reaching the optimal location. However, in the first case, it is desirable to perform the dissection at the optimal location, as more dissections can increase the risk of bleeding. Surgical visualization systems, as disclosed herein, can minimize the number of dissections by indicating the correct blood vessels and the optimal location for dissection. For example, the ureter and cardinal ligament are densely packed and present unique challenges during dissection. In such cases, minimizing the number of dissections may be particularly desirable.

[0449] In another example, a clinician (e.g., a surgical oncologist removing cancerous tissue) may want to know the identification of key structures, the location of the cancer, the stage of the cancer, and / or an assessment of tissue health. This type of information goes beyond what a clinician can see with the naked eye. Surgical visualization systems, as disclosed herein, can identify and / or communicate this information to the clinician intraoperatively to enhance intraoperative decision-making and improve surgical outcomes. In some cases, surgical visualization systems may be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, such as those using endoscopy or exoscopy.

[0450] In another example, a clinician (e.g., a surgical oncologist) might want to disable one or more alerts regarding the proximity of surgical instruments to one or more critical structures to avoid being overly conservative during surgery. In other cases, clinicians may want to receive certain types of alerts, such as tactile feedback (e.g., vibration / beep), to indicate proximity and / or “no-fly zones” to maintain adequate distance from one or more critical structures. For example, surgical visualization systems, as disclosed herein, can provide flexibility based on the clinician’s experience and / or the expected aggressiveness of the procedure. In such cases, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. Surgical visualization systems can aid in planning subsequent steps during surgery.

[0451] Various aspects of the subject matter described herein are set forth in the following numbered embodiments:

[0452] Example 1. A surgical system for surgical procedures, the surgical system comprising a surgical visualization system and control circuitry, the control circuitry being configured to provide a portion of an organ to be removed based on visualization data from the surgical visualization system, wherein the removal of the portion is configured to produce an estimated volume reduction of the organ, a first value of a non-visual parameter of the organ is determined before the portion is removed, and a second value of a non-visual parameter of the organ is determined after the portion is removed.

[0453] Example 2. The surgical system according to Example 1, wherein the non-visual parameters include peak lung volume measured by a ventilator.

[0454] Example 3. The surgical system according to Example 1, wherein the non-visual parameters include PCO2 measured by a ventilator.

[0455] Example 4. A surgical system according to any one of Examples 1 to 3, wherein the control circuit is further configured to confirm the estimated capacity reduction based on the first value of the non-visual parameter and the second value of the non-visual parameter.

[0456] Example 5. The surgical system according to Example 4, wherein the estimated volume reduction is equal to or less than a predetermined threshold.

[0457] Example 6. A surgical system according to any one of Examples 1 to 5, wherein the organ comprises a lung.

[0458] Example 7. The surgical system according to Example 6, wherein the control circuit is further configured to detect air leakage in the lung after the portion has been removed.

[0459] Example 8. The surgical system according to Example 7, wherein, after resection, the air leak is detected using dynamic visualization data of the lung from the visualization system.

[0460] Example 9. A surgical system for performing surgery, the surgical system comprising a surgical visualization system and control circuitry configured to receive input from a user indicating a portion of an organ to be removed, and to estimate the amount of volume reduction of the organ due to the removal of the portion based on visualization data from the surgical visualization system.

[0461] Example 10. The surgical system according to Example 9, wherein the control circuit is further configured to determine a first value of a non-visual parameter of the organ before the portion is removed.

[0462] Example 11. The surgical system according to Example 10, wherein the control circuit is further configured to determine a second value of the non-visual parameter of the organ after the portion has been removed.

[0463] Example 12. The surgical system according to Example 11, wherein the control circuit is further configured to confirm the estimated volume reduction of the organ based on the first value of the non-visual parameter and the second value of the non-visual parameter.

[0464] Example 13. The surgical system according to any one of Examples 10 to 12, wherein the non-visual parameters include peak lung volume measured by a ventilator.

[0465] Example 14. The surgical system according to any one of Examples 10 to 12, wherein the non-visual parameters include PCO2 measured by a ventilator.

[0466] Example 15. The surgical system according to any one of Examples 9 to 14, wherein the organ comprises a lung.

[0467] Example 16. The surgical system according to Example 15, wherein the control circuit is further configured to detect air leakage in the lung using dynamic visualization data of the lung from the visualization system after resection.

[0468] Example 17. A surgical system for surgical procedures, the surgical system comprising a surgical visualization system and a control circuit, the control circuit being configured to receive first visualization data from the surgical visualization system in a first state of an organ, determine a first value of a non-visualization parameter of the organ in the first state, receive second visualization data from the surgical visualization system in a second state of the organ, determine a second value of the non-visualization parameter of the organ in the second state, and detect tissue abnormalities based on the first visualization data, the second visualization data, the first value of the non-visualization parameter, and the second value of the non-visualization parameter.

[0469] Example 18. The surgical system according to Example 17, wherein the non-visual parameters include lung pressure measured by a ventilator.

[0470] Example 19. The surgical system according to Example 17, wherein the non-visual parameters include lung volume measured by a ventilator.

[0471] Example 20. A surgical system according to any one of Examples 17 to 19, wherein the first visualization data and the second visualization data include surface geometry measured by the surgical visualization system.

[0472] Example 21. A surgical system according to any one of Examples 17 to 20, wherein the organ comprises a lung, wherein the first state comprises the lung being inflated, and wherein the second state comprises the lung being contracted.

[0473] Although several forms have been illustrated and described, the applicant does not intend to limit or restrict the scope of the appended claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be made without departing from the scope of this disclosure, and those skilled in the art will recognize such modifications, variations, alterations, substitutions, combinations, and equivalents. Furthermore, alternatively, the structure of each element associated with a described form can be described as a device for providing the function performed by said element. Addit...

Claims

1. A surgical system for use in surgical procedures, the surgical system comprising: Surgical visualization system; and Control circuit, the control circuit being configured to: Based on visualization data from the surgical visualization system, the type of surgery to be performed, and the maximum expected volume to be removed, the portion of the organ to be removed is given, wherein the removal of the portion is configured to produce a reduction in the volume of the organ. Determine a first value for a non-visual parameter of the organ before removing the portion; and A second value for the non-visual parameter of the organ is determined after the portion is removed.

2. The surgical system according to claim 1, wherein, The non-visual parameters include peak lung volume measured by a ventilator.

3. The surgical system according to claim 1, wherein, The non-visual parameters include PCO2 measured by the ventilator.

4. The surgical system according to claim 1, wherein, The control circuit is further configured to verify the amount of capacity reduction based on the first value of the non-visual parameter and the second value of the non-visual parameter.

5. The surgical system according to claim 4, wherein, The reduction in capacity is equal to or less than a predetermined threshold.

6. The surgical system according to claim 1, wherein, The organs mentioned include the lungs.

7. The surgical system according to claim 6, wherein, The control circuit is further configured to detect air leakage in the lungs after the portion has been removed.

8. The surgical system according to claim 7, wherein, After the resection, the air leak was detected using dynamic visualization data of the lungs from the visualization system.

Images