Surgical systems that propose and support the removal of a portion of an organ

Abstract

To provide certain hidden structures, physical contours, and / or dimensions in three-dimensional space which provide information that cannot be recognized intraoperatively by certain imaging systems.SOLUTION: The surgical system includes a surgical visualization system and control circuitry configured to estimate, on the basis of visualization data from the surgical visualization system, at least one of a surface area and a volume of a lung with no portion resected, and estimate, on the basis of the estimated at least one of the surface area and the volume of the lung, a first maximum lung volume of the lung with no portion resected, estimate a second maximum lung volume of the lung after a portion of the lung is resected by a resection path determined on the basis of the visualization data from the surgical visualization system, and suggest the resection path when a ratio of the second maximum lung volume to the first maximum lung volume is greater than or equal to a predetermined value.SELECTED DRAWING: None

Description

[Background technology]

[0001] Surgical systems often incorporate imaging systems that allow one or more clinicians to view the surgical site and / or one or more parts thereof on one or more displays, such as monitors. These displays may be localized to the surgical theater and / or remote. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display visible to the clinician. Examples of scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or communicated to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. Additionally, certain imaging systems may be unable to communicate and / or transmit certain information to one or more clinicians during surgery. [Overview of the project] [Means for solving the problem]

[0002] In one general embodiment, a surgical system for use in surgical procedures is disclosed. This surgical system includes a surgical visualization system and a control circuit configured to, based on visualization data from the surgical visualization system, propose a portion of an organ to be resected, determine a first value of the non-visibility parameter of the organ before the partial resection, and determine a second value of the non-visibility parameter of the organ after the partial resection. The partial resection is configured to result in an estimated reduction in the organ's volume.

[0003] In another general aspect, a surgical system for use in surgery is disclosed. The surgical system includes a surgical visualization system and a control circuit configured to receive an input from a user indicating a portion of an organ to be resected based on visualization data from the surgical visualization system and to estimate a reduction in the volume of the organ by removing the portion.

[0004] In yet another general aspect, a surgical system for use in surgery is disclosed. The surgical system includes a surgical visualization system and a control circuit configured to receive first visualization data of an organ in a first state from the surgical visualization system, determine a first value of a non-visualization parameter of the organ in the first state, receive second visualization data of the organ in a second state from the surgical visualization system, determine a second value of the non-visualization parameter of the organ in the second state, and detect tissue abnormalities based on the first visualization data, the second visualization data, the first value of the non-visualization parameter, and the second value of the non-visualization parameter.

Brief Description of the Drawings

[0005] The novel features of the various aspects are set forth specifically in the appended "Claims". However, the described aspects may be best understood by reference to the following description taken in conjunction with the accompanying drawings in terms of both their organization and method of operation. [Figure 1] A schematic diagram of a surgical visualization system including an imaging device and a surgical device according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify important structures below the surface of tissue. [Figure 2] A schematic diagram of a control system of a surgical visualization system according to at least one aspect of the present disclosure. [Figure 2A] A control circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown. [Figure 2B]Shows a combinational logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure. [Figure 2C] Shows a sequential logic circuit configured to control aspects of a surgical visualization system according to at least one aspect of the present disclosure. [Figure 3] Schematic diagram showing triangulation among the surgical device, imaging device, and critical structure of FIG. 1 for determining the depth da of a critical structure below the tissue surface according to at least one aspect of the present disclosure. [Figure 4] Schematic diagram of a surgical visualization system configured to identify critical structures below the tissue surface according to at least one aspect of the present disclosure, the surgical visualization system including a pulsed light source for determining the depth da of critical structures below the tissue surface. [Figure 5] Schematic diagram of a surgical visualization system including an imaging device and a surgical device according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify critical structures below the tissue surface. [Figure 6] Schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify critical structures embedded within the tissue. [Figure 7A] View of a critical structure photographed by the three-dimensional camera of FIG. 6 according to at least one aspect of the present disclosure, FIG. 7A being a view from the left lens of the three-dimensional camera and FIG. 7B being a view from the right lens of the three-dimensional camera. [Figure 7B] View of a critical structure photographed by the three-dimensional camera of FIG. 6 according to at least one aspect of the present disclosure, FIG. 7A being a view from the left lens of the three-dimensional camera and FIG. 7B being a view from the right lens of the three-dimensional camera. [Figure 8] Schematic diagram of the surgical visualization system of FIG. 6 capable of determining the camera-critical structure distance dw from the three-dimensional camera to the critical structure according to at least one aspect of the present disclosure. [Figure 9]This is a schematic diagram of a surgical visualization system that utilizes two cameras to determine the location of an implanted critical structure, relating to at least one aspect of the present disclosure. [Figure 10A] This is a schematic diagram of a surgical visualization system, according to at least one aspect of the present disclosure, which utilizes a camera that is moved axially between a plurality of known locations to determine the location of an embedded critical structure. [Figure 10B] Figure 10A is a schematic diagram of a surgical visualization system relating to at least one aspect of the present disclosure, in which a camera is moved axially and rotationally between a plurality of known locations to determine the location of an embedded critical structure. [Figure 11] This is a schematic diagram of a control system for a surgical visualization system relating to at least one aspect of the present disclosure. [Figure 12] This is a schematic diagram of a structured light source for a surgical visualization system relating to at least one aspect of the present disclosure. [Figure 13A] This figure shows a graph of absorption coefficient versus wavelength for various biomaterials relating to at least one aspect of this disclosure. [Figure 13B] This is a schematic diagram of visualization of anatomical structures by a spectral surgical visualization system according to at least one aspect of the present disclosure. [Figure 13C] Figure 13C shows exemplary hyperspectral identifying signatures for differentiating anatomical structures from occlusions, relating to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus occlusion, Figure 13D is a graph representation of an arterial signature versus occlusion, and Figure 13E is a graph representation of a nerve signature versus occlusion. [Figure 13D] Figure 13C shows exemplary hyperspectral identifying signatures for differentiating anatomical structures from occlusions, relating to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus occlusion, Figure 13D is a graph representation of an arterial signature versus occlusion, and Figure 13E is a graph representation of a nerve signature versus occlusion. [Figure 13E]Figure 13C shows exemplary hyperspectral identifying signatures for differentiating anatomical structures from occlusions, relating to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus occlusion, Figure 13D is a graph representation of an arterial signature versus occlusion, and Figure 13E is a graph representation of a nerve signature versus occlusion. [Figure 14] This is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to detect the distance to an important anatomical structure, relating to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) located on a common device. [Figure 15] This is a schematic diagram of the radiated wave, received wave, and delay between the radiated wave and the received wave of the NIR time-of-flight measurement system of Figure 17A, relating to at least one aspect of the present disclosure. [Figure 16] This disclosure describes at least one aspect of a NIR time-of-flight measurement system configured to detect distances to different structures, the time-of-flight measurement system comprising a transmitter (emitter) and a receiver (sensor) on separate devices. [Figure 17] This is a block diagram of a computer-implemented interactive surgical system relating to at least one aspect of the present disclosure. [Figure 18] This is a diagram of a surgical system used to perform surgical procedures in an operating room, relating to at least one aspect of the present disclosure. [Figure 19] This document describes a computer-implemented interactive surgical system relating to at least one aspect of the present disclosure. [Figure 20] A diagram of a situational awareness surgical system relating to at least one aspect of this disclosure is shown. [Figure 21] A timeline illustrating the hub's situational awareness relating to at least one aspect of this disclosure is shown. [Figure 22] This is a logical flow diagram of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for correlating visualization data with instrument data. [Figure 23]This is a schematic diagram of a surgical instrument relating to at least one aspect of the present disclosure. [Figure 24] This figure shows a composite dataset and a graph illustrating virtual gauges for Force-to-Close (FTC) and Force-to-Fire (FTF), relating to at least one aspect of the present disclosure. [Figure 25A] This shows a standard view of the screen of a visualization system that displays live video of an end effector in the surgical field during surgery, relating to at least one aspect of this disclosure. [Figure 25B] This shows an extended view of a screen of a visualization system that displays live video of an end effector in the surgical field during a surgical procedure, relating to at least one aspect of this disclosure. [Figure 26] This is a logical flow diagram of a process showing a control program or logical configuration for synchronizing the virtual display movement of an end effector component with the actual movement of the end effector component, relating to at least one aspect of the present disclosure. [Figure 27] An at least one aspect of the present disclosure shows a body wall and anatomical structures in a cavity below the body wall, the trocar entering the cavity through the body wall, and the screen displays the distance of the trocar from the anatomical structure, the risks associated with presenting surgical instruments through the trocar, and the estimated surgical time associated therewith. [Figure 28] This shows a virtual three-dimensional ("3D") structure of a stomach exposed to structured light from a structured light projector, relating to at least one aspect of the present disclosure. [Figure 29] This is a logical flow diagram of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for correlating visualization data with instrument data, where dashed boxes indicate alternative implementations of the process. [Figure 30] This shows a virtual 3D structure of a stomach exposed to structured light from a structured light projector, relating to at least one aspect of this disclosure. [Figure 31]This is a logical flowchart of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for proposing a resection route for removing a portion of an anatomical organ, where dashed boxes indicate alternative implementations of the process. [Figure 32A] This shows a live view of the surgical field on a screen of a visualization system at the start of a surgical procedure, relating to at least one aspect of this disclosure. [Figure 32B] This is an enlarged view of a portion of the surgical field in Figure 32A, outlining a proposed surgical resection route superimposed on the surgical field, relating to at least one aspect of this disclosure. [Figure 32C] Figure 32B shows a live view of the surgical field 43 minutes after the start of a surgical procedure, relating to at least one aspect of this disclosure. [Figure 32D] Figure 32C shows an enlarged view of the surgical field, outlining the proposed modification of the surgical resection route according to at least one aspect of this disclosure. [Figure 33] This is a logical flowchart of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for presenting parameters of a surgical instrument on or near a proposed surgical resection path, where dashed boxes indicate alternative implementations of the process. [Figure 34] This shows a virtual 3D structure of the stomach of a patient undergoing sleeve gastrectomy, according to at least one aspect of this disclosure. [Figure 35] Figure 34 shows a hypothetical resection of the stomach after completion. [Figure 36A] This shows the firing of a surgical stapling instrument according to at least one aspect of the present disclosure. [Figure 36B] This shows the firing of a surgical stapling instrument according to at least one aspect of the present disclosure. [Figure 36C] This shows the firing of a surgical stapling instrument according to at least one aspect of the present disclosure. [Figure 37] This is a logical flow diagram of a process showing a control program or logical configuration for adjusting the firing speed of a surgical instrument, relating to at least one aspect of the present disclosure. [Figure 38] This is a logical flowchart of a process illustrating a control program or logical configuration for proposed staple cartridge placement along a proposed surgical resection route, relating to at least one aspect of the present disclosure. [Figure 39] This is a logical flowchart illustrating a process for illustrating a control program or logical configuration for proposing the surgical resection of a portion of an organ, relating to at least one aspect of the present disclosure. [Figure 40] This is a logical flowchart of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for estimating the reduction in organ volume resulting from the removal of a selected portion of an organ. [Figure 41A] A patient's lung exposed to structured light, including a portion to be resected during surgery, as per at least one aspect of this disclosure. [Figure 41B] Figure 41A shows the lung of a patient after partial resection, relating to at least one aspect of this disclosure. [Figure 41C] The graphs relating to at least one aspect of this disclosure show how to measure the maximum lung volume of the patient in Figures 41A and 41B before and after partial lung resection. [Figure 42] The graph shows, according to at least one aspect of this disclosure, the partial pressure of carbon dioxide (PCO2) in a patient's lung before, immediately after, and one minute after partial lung resection. [Figure 43] This is a logical flow diagram of a process relating to at least one aspect of the present disclosure, showing a control program or logical configuration for detecting tissue anomalies using visualized and non-visible data. [Figure 44A] The right lung in a first state according to at least one aspect of the present disclosure is shown with an imaging device illuminating its surface with a light pattern. [Figure 44B] The right lung of Figure 44 in a second state according to at least one aspect of this disclosure is shown with the imaging device illuminating its surface with a light pattern. [Figure 44C] Figure 44A shows the apex of the right lung according to at least one aspect of the present disclosure. [Figure 44D] Figure 44B shows the apex of the right lung according to at least one aspect of the present disclosure. [Modes for carrying out the invention]

[0006] The applicant of this application also owns the following concurrently filed U.S. patent applications, each of which is incorporated herein by reference in its entirety. • A document titled "METHOD OF USING IMAGING DEVICES IN SURGERY," with agent reference number END9228USNP1 / 190580-1M; • Titled "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM," Agent Reference Number END9227USNP1 / 190579-1; • Agent reference number END9226USNP1 / 190578-1, titled "SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS"; • Titled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS," Agent Reference Number END9225USNP1 / 190577-1; • Titled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS," Agent Reference Number END9224USNP1 / 190576-1; • Titled "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA," Agent Reference Number END9223USNP1 / 190575-1; • Titled "SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED," Agent Reference Number END9222USNP1 / 190574-1; • Titled "SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN," Agent Reference Number END9221USNP1 / 190573-1; • Agent reference number END9219USNP1 / 190571-1, titled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE"; • Titled "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT," Agent Reference Number END9218USNP1 / 190570-1; • Titled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS," Agent Reference Number END9217USNP1 / 190569-1; • Titled "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM," agent reference number END9216USNP1 / 190568-1.

[0007] The applicant of this application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety. U.S. Patent Application No. 16 / 354,417, entitled "INPUT CONTROLS FOR ROBOTIC SURGERY"; U.S. Patent Application No. 16 / 354,420, titled "DUAL MODE CONTROLS FOR ROBOTIC SURGERY"; • U.S. Patent Application No. 16 / 354,422, entitled "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY"; U.S. Patent Application No. 16 / 354,440, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY"; U.S. Patent Application No. 16 / 354,444, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE"; • U.S. Patent Application No. 16 / 354,454, entitled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS"; U.S. Patent Application No. 16 / 354,461, entitled "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS"; U.S. Patent Application No. 16 / 354,470, entitled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS"; • U.S. Patent Application No. 16 / 354,474, entitled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES"; • U.S. Patent Application No. 16 / 354,478, titled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK", and U.S. Patent Application No. 16 / 354,481, titled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS".

[0008] The applicant of this application also owns the following U.S. patent applications filed on September 11, 2018, the entire contents of each of these applications incorporated herein by reference. • U.S. Patent Application No. 16 / 128,179, titled "SURGICAL VISUALIZATION PLATFORM"; U.S. Patent Application No. 16 / 128,180, entitled "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE"; • U.S. Patent Application No. 16 / 128,198, titled "SINGULAR EMR SOURCE EMITTER ASSEMBLY"; • U.S. Patent Application No. 16 / 128,207, titled "COMBINATION EMITTER AND CAMERA ASSEMBLY"; U.S. Patent Application No. 16 / 128,176, entitled "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES"; U.S. Patent Application No. 16 / 128,187, entitled "SURGICAL VISUALIZATION OF MULTIPLE TARGETS"; • U.S. Patent Application No. 16 / 128,192, titled "VISUALIZATION OF SURGICAL DEVICES"; • U.S. Patent Application No. 16 / 128,163, titled "OPERATIVE COMMUNICATION OF LIGHT"; • U.S. Patent Application No. 16 / 128,197, titled "ROBOTIC LIGHT PROJECTION TOOLS"; • U.S. Patent Application No. 16 / 128,164, entitled "SURGICAL VISUALIZATION FEEDBACK SYSTEM"; • U.S. Patent Application No. 16 / 128,193, entitled "SURGICAL VISUALIZATION AND MONITORING"; • U.S. Patent Application No. 16 / 128,195, titled "INTEGRATION OF IMAGING DATA"; • U.S. Patent Application No. 16 / 128,170, entitled "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS"; • U.S. Patent Application No. 16 / 128,183, entitled "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS"; • U.S. Patent Application No. 16 / 128,172, entitled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER"; U.S. Patent Application No. 16 / 128,185, titled "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION".

[0009] The applicant of this application also owns the following U.S. patent applications filed on March 29, 2018, the entire contents of each of these applications incorporated herein by reference. U.S. Patent Application No. 15 / 940,627, entitled "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," currently published as U.S. Patent Application Publication No. 2019 / 0201111; U.S. Patent Application No. 15 / 940,676, entitled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," currently published as U.S. Patent Application Publication No. 2019 / 0201142; U.S. Patent Application No. 15 / 940,711, entitled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," currently published as U.S. Patent Application Publication No. 2019 / 0201120; U.S. Patent Application No. 15 / 940,722, titled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY," currently published as U.S. Patent Application Publication No. 2019 / 0200905.

[0010] The applicant of this application owns the following U.S. patent applications filed on December 4, 2018, the disclosures of which are incorporated herein by reference in their entirety. U.S. Patent Application No. 16 / 209,395, entitled "METHOD OF HUB COMMUNICATION," currently published as U.S. Patent Application Publication No. 2019 / 0201136; • U.S. Patent Application No. 16 / 209,403, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," currently published as U.S. Patent Application Publication No. 2019 / 0206569; U.S. Patent Application No. 16 / 209,407, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," currently published as U.S. Patent Application Publication No. 2019 / 0201137; U.S. Patent Application No. 16 / 209,416, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS," currently published as U.S. Patent Application Publication No. 2019 / 0206562; U.S. Patent Application No. 16 / 209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS," currently published as U.S. Patent Application Publication No. 2019 / 0200981; U.S. Patent Application No. 16 / 209,427, entitled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES," currently published as U.S. Patent Application Publication No. 2019 / 0208641; U.S. Patent Application No. 16 / 209,433, entitled "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB," currently published as U.S. Patent Application Publication No. 2019 / 0201594; U.S. Patent Application No. 16 / 209,447, entitled "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB," currently published as U.S. Patent Application Publication No. 2019 / 0201045; U.S. Patent Application No. 16 / 209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES," currently published as U.S. Patent Application Publication No. 2019 / 0201046; U.S. Patent Application No. 16 / 209,458, entitled "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE," currently published as U.S. Patent Application Publication No. 2019 / 0201047; U.S. Patent Application No. 16 / 209,465, entitled "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION," currently published as U.S. Patent Application Publication No. 2019 / 0206563; U.S. Patent Application No. 16 / 209,478, titled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE," currently published as U.S. Patent Application Publication No. 2019 / 0104919; • U.S. Patent Application No. 16 / 209,490, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION," currently published as U.S. Patent Application Publication No. 2019 / 0206564; U.S. Patent Application No. 16 / 209,491, entitled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS," currently published as U.S. Patent Application Publication No. 2019 / 0200998.

[0011] Before describing in detail the various embodiments of the surgical visualization platform, it should be noted that the exemplary embodiments are not limited in their application or use to the structural and arrangement details of the components illustrated in the accompanying drawings and specification. The exemplary embodiments may be implemented or incorporated into other embodiments, variations, and modifications, and may be carried out or performed in various ways. Furthermore, unless otherwise specified, the terms and expressions used herein have been selected for the purpose of describing the exemplary embodiments for the convenience of the reader and are not intended to limit them. Furthermore, it should be understood that one or more embodiments, expressions of embodiments, and / or embodiments described below may be combined with any one or more other embodiments, expressions of embodiments, and / or embodiments described below.

[0012] Surgical visualization system This disclosure pertains to a surgical visualization platform that leverages “digital surgery” to obtain additional information about a patient’s anatomical structure and / or surgical procedures. The surgical visualization platform is further configured to communicate the data and / or information to one or more clinicians in a useful format. For example, various aspects of this disclosure provide improved visualization of a patient’s anatomical structure and / or surgical procedures.

[0013] "Digital surgery" may encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, and connectivity both inside and outside the operating room (OR). The various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, but are not limited to use with robotic surgical systems. In certain examples, advanced surgical visualization can be performed without a robot, and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot, and / or with limited and / or optional robotic assistance.

[0014] In certain cases, surgical systems incorporating a surgical visualization platform can enable smart incisions to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries (including superior mesenteric arteries), veins (including portal veins), nerves (including phrenic nerves), and / or tumors, among other anatomical structures. In other cases, critical structures may be foreign body structures in an anatomical area, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, and / or plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Illustrative critical structures are further described herein. Smart incisions can provide improved intraoperative guidance for incisions and / or enable smarter decision-making, for example, through techniques for detecting and avoiding critical anatomical structures.

[0015] Surgical systems incorporating surgical visualization platforms can also enable smart anastomosis, which provides more consistent anastomoses at the optimal location(s) through improved workflows. Cancer localization methods can also be improved by the various surgical visualization platforms and procedures described herein. For example, cancer localization methods can identify and track the location, orientation, and margins of a tumor. In certain cases, cancer localization methods can compensate for the movement of instruments, patients, and / or patient anatomical structures during surgery, and guide the clinician back to the point of interest.

[0016] In certain aspects of this disclosure, the surgical visualization platform may provide improved tissue characterization, and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize the type and health of tissue without requiring physical touch, particularly during incision and / or placement of stapling devices within the tissue. Certain tissue characterization techniques described herein can be used without the use of ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can, for example, locate, map, and ideally diagnose lymphatic systems and / or lymph nodes involved in cancer diagnosis and staging before surgery.

[0017] During surgery, the information available to the clinician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures, such as those embedded or buried within organs, may be at least partially hidden, i.e., invisible, from the view. Additionally, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or difficult to grasp with the "naked eye." Furthermore, certain structures may move before surgery (e.g., before surgery but after preoperative scanning) and / or during surgery. In such cases, the clinician may not be able to accurately determine the location of critical structures during surgery.

[0018] When the location of critical structures is uncertain, and / or the proximity of critical structures to surgical instruments is unknown, the clinician's decision-making process can be hindered. For example, a clinician may avoid certain areas to avoid inadvertently cutting a critical structure. However, the avoided area may be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or excessive caution, a clinician may fail to reach a specific desired area. For example, even if a critical structure is not in that particular area, and / or the clinician's actions in that area would have no adverse effects, over-caution may cause the clinician to try to avoid the critical structure, leaving behind a portion of the tumor and / or other undesirable tissue. In certain cases, surgical outcomes may improve with increased knowledge and / or certainty, which allows surgeons to be more precise and, in certain cases, more restrained / more aggressive towards specific anatomical areas.

[0019] In various embodiments, the Disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one embodiment, the Disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various embodiments, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what a clinician sees with the naked eye and / or what an imaging system can perceive and / or communicate to the clinician. Various surgical visualization systems can improve outcomes in various cases by reinforcing and enhancing what a clinician can know before tissue treatment (e.g., incision).

[0020] For example, a visualization system may include a first optical emitter configured to emit multiple spectral waves, a second optical emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to spectral waves (spectroscopic imaging), and / or the light pattern. Throughout the following disclosure, unless specifically referring to visible light, all references to “light” may include electromagnetic waves or photons in the visible and / or invisible portions of the electromagnetic radiation (EMR) wavelength spectrum. A surgical visualization system may also include an imaging system and a control circuit that signals the receiver(s) and the imaging system. Based on the output from the receiver(s), the control circuit may determine a geometric surface map of the visible surface at the surgical site, i.e., a three-dimensional surface topography, and one or more distances to the surgical site. In certain examples, the control circuit may determine one or more distances to at least partially hidden structures. Furthermore, the imaging system may communicate the geometric surface map and one or more distances to the clinician. In such examples, an augmented view of the surgical site provided to the clinician can provide a display of hidden structures within the context of the surgical site. For example, the imaging system can virtually enhance hidden structures on a geometric surface map of the hidden and / or obscured tissue, as well as lines drawn on the ground to indicate utility piping below the surface. Additionally, or alternatively, the imaging system can communicate the proximity of one or more surgical instruments to the visible obscured tissue and / or to at least partially hidden structures, as well as the depth of structures hidden below the visible surface of the obscured tissue. For example, the visualization system can determine the distance to an augmented line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of this disclosure, surgical visualization systems for intraoperative identification and avoidance of critical structures are disclosed. Such surgical visualization systems can provide clinicians with useful information during surgical procedures. As a result, clinicians can ensure they maintain momentum throughout the surgery while recognizing that the surgical visualization system is tracking critical structures, such as ureters, specific nerves, and / or important blood vessels, that may be approached during incisions. In one aspect, the surgical visualization system can provide clinicians with sufficient time to pause and / or slow down the surgery and assess their proximity to critical structures to prevent inadvertent damage to those structures. The surgical visualization system can provide clinicians with an ideal, optimized, and / or customizable amount of information, enabling them to move safely and / or quickly across entire tissues while avoiding inadvertent damage to healthy tissues and / or critical structures(s), thereby minimizing the risk of injury resulting from the surgery.

[0022] Figure 1 is a schematic diagram of a surgical visualization system 100 relating to at least one aspect of the present disclosure. The surgical visualization system 100 can produce a visual representation of critical structures 101 within an anatomical region. The surgical visualization system 100 can be used, for example, in clinical analysis and / or medical intervention. In certain examples, the surgical visualization system 100 can be used during surgery to provide clinicians with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during surgical procedures. The surgical visualization system 100 is configured for the intraoperative identification of critical structures(s) and / or to facilitate avoidance of critical structures(s) 101 by surgical devices. For example, by identifying critical structures 101, clinicians can avoid manipulating surgical devices around critical structures 101 and / or a predetermined proximal region of critical structures 101 during surgical procedures. Clinicians can avoid incisions of, for example, veins, arteries, nerves, and / or blood vessels, and / or vicinity thereof, which are identified as critical structures 101. In various cases, the critical structure 101 may be determined on a patient-by-patient and / or treatment-by-treatment basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. When combined, these features of the surgical visualization system 100 can determine the location of critical structures 101 within an anatomical region, and / or the proximity of surgical devices 102 to the surface of visible tissue 105 and / or critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of the image based on molecular responses to various wavelengths. The view from the imaging device 120 can be provided to a clinician and, in various aspects of this disclosure, can be enhanced with additional information based on tissue identification, landscape mapping, and the distance sensor system 104. In such an example, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide the clinician(s) with advanced data synthesis and integrated information during surgery.

[0024] The imaging device may include, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In various aspects of this disclosure, the imaging system may include, for example, an imaging device such as an endoscope. Additionally or alternatively, the imaging system may include, for example, an imaging device such as an arthroscope, angioscope, bronchoscope, cholangioscope, coloscope, cystoscope, duodenoscope, intestinaloscope, esophagogastroduodenoscope (gastroscopy), laryngoscope, nasopharyngolaryngoscope, sigmoidoscopy, thoracoscopy, ureteroscope, or exoscopy. In other examples, such as in open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of this disclosure, the tissue identification subsystem can be achieved using a spectral imaging system. The spectral imaging system may rely, for example, on hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, issued March 1, 2016, entitled "System and method for gross anatomic pathology using hyperspectral imaging," which is incorporated herein by reference in its entirety.

[0026] In various aspects of this disclosure, the surface mapping subsystem can be achieved using a light pattern system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structured light is the process of projecting a known pattern (often a grid or horizontal bars) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, “SET COMPRISING A SURGICAL INSTRUMENT,” published March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, “DEPICTION SYSTEM,” published September 7, 2017, disclose a surgical system comprising a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, “SET COMPRISING A SURGICAL INSTRUMENT,” and U.S. Patent Application Publication No. 2017 / 0251900, “DEPICTION SYSTEM,” are incorporated herein by reference to their entirety.

[0028] In various aspects of this disclosure, the distance determination system can be incorporated into a surface mapping system. For example, structured light can be used to generate a three-dimensional virtual model of a visible surface and determine various distances to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to specific tissues (or other structures) at a surgical site.

[0029] Figure 2 is a schematic diagram of a control system 133 that can be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 that communicates with a memory 134. The memory 134 stores instructions that can be executed by the control circuit 132 for determining and / or recognizing important structures (e.g., important structure 101 in Figure 1), determining and / or calculating one or more distances and / or three-dimensional digital displays, and communicating specific information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (such as the imaging device 120 in Figure 1), one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 may include one or more image sensors 135 (in particular, for example, a visible light, a spectral imager, a three-dimensional lens) for receiving signals from various light sources that emit light in various visible and invisible spectra. The display 146 may include one or more screens or monitors for displaying real, virtual, and / or virtually augmented images and / or information to one or more clinicians.

[0030] In various embodiments, the central component of the camera 144 is the image sensor 135. Generally, modern image sensors 135 are solid-state electronic devices containing up to several million separate photodetector segments called pixels. Image sensor 135 technology is classified into one of two categories: charge-coupled device (CCD) imagers and capture-type metal-oxide-semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being a more recent development in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (marketed as "sCOMS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. Both CCD and CMOS image sensors 135 have sensitivity to wavelengths of approximately 350–1050 nm, although this range is typically 400–1000 nm. CMOS sensors are generally more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Therefore, there are two types of color CCD cameras: 1-chip and 3-chip. 1-chip color CCD cameras offer a commonly adopted, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to separate incident light into a series of colors and employing interpolation algorithms to resolve a full-color image. Each color is then directed to a different set of pixels. 3-chip color CCD cameras offer higher resolution by employing prisms, directing each section of the incident spectrum to a different chip. Because each point in the object's space has distinct RGB intensity values ​​rather than using an algorithm to determine color, more accurate color reproduction is possible. 3-chip cameras offer extremely high resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structured light source 152. In certain examples, a single light source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, a single light source can be pulsed to supply light in the visible spectrum (e.g., infrared spectral light) and wavelengths of light on the visible spectrum. The spectral light source 150 may be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various examples, tissue identification logic 140 can identify one or more important structures via data from the spectral light source 150 received by the image sensor portion 135 of the camera 144. Surface mapping logic 136 can determine the surface contour of visible tissue based on reflected structured light. By time-of-flight measurement, distance determination logic 141 can determine one or more distances to visible tissue and / or important structures 101. One or more outputs from the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 may be provided to the imaging logic 138 and combined, integrated, and / or superimposed so as to be communicated to the clinician via the display 146 of the imaging system 142.

[0032] Figures 2A to 2C will now be briefly described to illustrate various aspects of the control circuit 132 for controlling various aspects of the surgical visualization system 100. Referring to Figure 2A, a control circuit 400 is shown which is configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The control circuit 400 can be configured to implement various processes described herein. The control circuit 400 may include a microcontroller having one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions which, when executed by the processor 402, cause the processor 402 to execute machine instructions for implementing various processes described herein. The processor 402 may be any one of the many single-core or multi-core processors known in the art. The memory circuit 404 may include volatile and non-volatile storage media. The processor 402 may also include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0033] Figure 2B shows a combinational logic circuit 410 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 may include a finite state machine comprising a combinational logic 412 configured to receive data associated with a surgical instrument or surgical device at an input 414, process the data by combinational logic 412, and supply an output 416.

[0034] Figure 2C shows a sequential logic circuit 420 configured to control an aspect of a surgical visualization system 100 according to at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may comprise a finite state machine. The sequential logic circuit 420 may comprise, for example, a combinational logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store the current state of the finite state machine. In certain examples, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic 422 is configured to receive data associated with a surgical device or system from an input 426, process the data by the combinational logic 422, and provide an output 428. In other aspects, the circuit may comprise a combination of a processor (e.g., a processor 402 in Figure 2A) and a finite state machine that implements various processes described herein. In other embodiments, the finite state machine may comprise a combination of a combinational logic circuit (e.g., the combinational logic circuit 410 in Figure 2B) and a sequential logic circuit 420.

[0035] Referring again to the surgical visualization system 100 in Figure 1, the critical structure 101 may be an anatomical structure of the subject. For example, the critical structure 101 may be an artery such as the ureter or superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor, among other anatomical structures. In other examples, the critical structure 101 may be a foreign body structure in an anatomical area, such as a surgical device, surgical fastener, clip, clasp, bougie, band, and / or plate. Exemplary critical structures are described further herein and in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,192, filed September 11, 2018, entitled "VISUALIZATION OF SURGICAL DEVICES," which are each incorporated herein by reference in their entirety.

[0036] In one embodiment, the critical structure 101 may be embedded in tissue 103. In other words, the critical structure 101 may be located below the surface 105 of tissue 103. In such an example, tissue 103 hides the critical structure 101 from the clinician's view. The critical structure 101 is also shielded from the view of the imaging device 120 by tissue 103. Tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other examples, the critical structure 101 may be partially shielded from view.

[0037] Figure 1 also shows a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from the distal end of the shaft of the surgical device 102. The surgical device 102 may be any suitable surgical device, such as an incision instrument, stapler, gripping instrument, clip applicator, and / or an energy device including a unipolar probe, bipolar probe, ablation probe, and / or an ultrasonic end effector. Additionally or alternatively, the surgical device 102 may include another imaging or diagnostic mode, such as an ultrasonic device. In one aspect of the present disclosure, the surgical visualization system 100 may be configured to achieve the identification of one or more critical structures 101 and the surgical device 102's access to the critical structures (one or more) 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral camera, a multispectral camera, or a selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral image sensor, detector, and / or three-dimensional camera lens). For example, the imaging device 120 may include right and left lenses used together to generate a three-dimensional image of the surgical site by simultaneously recording two two-dimensional images, to render a three-dimensional image of the surgical site, and / or to determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images showing topography of visible tissue, as well as the identification and location of hidden important structures, as further described herein. For example, as shown in Figure 1, the field of view of the imaging device 120 can be superimposed on a light pattern (structured light) on the surface 105 of the tissue.

[0039] In one embodiment, the surgical visualization system 100 may be incorporated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 include rigid structural members 116 and joints 118 that can include servo motor control. The first robotic arm 112 is configured to operate a surgical device 102, and the second robotic arm 114 is configured to operate an imaging device 120. A robotic control unit can be configured to generate control movements to the robotic arms 112 and 114 that can act on the surgical device 102 and the imaging device 120, for example.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit light patterns such as stripes, grid lines, and / or dots to enable determination of the topography or landscape of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from an emitter 106 located on, for example, a surgical device 102 and / or one of the robotic arms 112, 114 and / or an imaging device 120. In one embodiment, the projected light array 130 is employed to determine the shape of the tissue 103 surface 105 and / or the shape defined during surgery by the movement of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0041] In one embodiment, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of tissue 103 to reach critical structures 101. The imaging device 120 and the optical waveform emitter 123 on it may be positionable by a robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted from the optical waveform emitter 123 may be configured to enable the identification of types of anatomical structures and / or body structures, such as critical structures 101. Identification of critical structures 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one embodiment, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 may include, for example, a photoacoustic imaging system. In other examples, the optical waveform emitter 123 may be located on a surgical device separate from the imaging device 120.

[0042] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one embodiment, the time-of-flight distance sensor system 104 may include an emitter such as an emitter 106 and a receiver 108 which may be located on the surgical device 102. In other examples, the time-of-flight emitter may be separate from a structured light emitter. In one common embodiment, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a matched sensor. The time-of-flight distance sensor system 104 can detect "time of flight," that is, the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, the distance sensor system 104 is able to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104. Referring further to Figure 1, d e d is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and t This is the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. A distance sensor system 104 is employed to measure the emitter-tissue distance d. e The device-organizational distance d can be determined. t This can be obtained from the known position of the emitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, then the device-tissue distance d t is the emitter-tissue distance d e This can be determined from the following. In certain cases, the shaft of the surgical device 102 may include one or more articular joints and may be articulate with respect to the emitter 106 and jaws. The articulation configuration may include, for example, a multi-articular vertebral-like structure. In certain cases, a three-dimensional camera can be used to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on a separate surgical device instead of the surgical device 102. For example, the receiver 108 can be mounted on a cannula or trocar that extends through the surgical device 102 to reach the surgical site. In yet another example, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on another robotic control arm (e.g., robotic arm 114), on another robotically controlled movable arm, and / or on an operating room (OR) table or fixture. In a particular example, the imaging device 120 includes a time-of-flight receiver 108 that uses a line between the emitter 106 on the surgical device 102 and the imaging device 120 to determine the distance from the emitter 106 to the surface 105 of the tissue 103. For example, based on the known positions of the emitter 106 of the time-of-flight distance sensor system 104 (on the surgical device 102) and the receiver 108 (on the imaging device 120), the distance d e The three-dimensional position of receiver 108 can be determined and / or aligned with the robot coordinate plane during surgery.

[0044] In certain examples, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other examples, the surgical visualization system 100 can be used separately from the robotic system. In such examples, the distance sensor system 104 may be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the main robotic system used in surgery. At least one of the robotic arms 112, 114 can be placed and aligned in a particular coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 can control and / or align the position of the robotic arm(s) 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned relative to a particular coordinate system.

[0046] Referring further to FIG. 1, d w is the camera-important structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the important structure 101, and d A is the depth of the important structure 101 below the surface 105 of the tissue 103 (i.e., the distance between a portion of the surface 105 closest to the surgical device 102 and the important structure 101). In various aspects, the flight time of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-important structure distance d w . The use of spectral imaging in combination with a flight time sensor is further described herein. Further, referring now to FIG. 3, in various aspects of the present disclosure, the depth d of the important structure 101 relative to the surface 105 of the tissue 103 A is the distance d w , and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the distance d between them x ), and can be determined by triangulation from the sum of the distances d e and d A to obtain the distance d y .

[0047] Additionally, or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, the first waveform (or waveform range) can be used to determine the camera-critical structure distance d w The second waveform (or waveform range) can be used to determine the distance from the surface 105 of the tissue 103. In such an example, different waveforms can be used to determine the depth of the important structure 101 below the surface 105 of the tissue 103.

[0048] Additionally, or alternatively, in certain examples, distance d A This can be determined from ultrasound, a registered magnetic resonance imaging (MRI) view, or a computed tomography (CT) scan. In yet another example, distance d A This can be determined by spectral imaging, as the detection signal received by the imaging device may vary depending on the type of material. For example, fat can reduce the detection signal by a first method or a first amount, while collagen can reduce the detection signal by a different second method or a second amount.

[0049] Referring here to the surgical visualization system 160 in Figure 4, the surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform emitter 123 is located at a distance d from common devices such as the surgical device 162, as will be further described herein. t , and d w It can be configured to emit a waveform for determining the distance d from the surface 105 of tissue 103 to the surface of critical structure 101. A This can be determined as follows: d A =d w -d t .

[0050] As disclosed herein, various information regarding visible tissue, implanted critical structures, and surgical devices can be determined by utilizing an approach that incorporates one or more time-of-flight distance sensors, spectral imaging, and / or structural light arrays, in combination with an image sensor configured to detect spectral wavelengths and structured light arrays. Furthermore, the image sensor may be configured to provide an image of the surgical site to the imaging system by receiving visible light. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and to render a three-dimensional image of the surface tissue and underlying anatomical structures. In various examples, the imaging device 120 may include multiple image sensors.

[0051] Camera-to-critical structure distance d w Furthermore, it can be detected by one or more alternative methods. In one embodiment, for example, a fluorescence visualization technique such as fluorescent indocyanine green (ICG) can be used to illuminate the important structure 201 as shown in Figures 6-8. The camera 220 may include two optical waveform sensors 222, 224 that simultaneously capture left and right images of the important structure 201 (Figures 7A and 7B). In such an example, the camera 220 can depict the emission of the important structure 201 below the surface 205 of the tissue 203, at a distance d w This can be determined by the known distance between sensors 222 and 224. In certain examples, the distance can be determined more accurately by using two or more cameras or by moving the cameras between multiple positions. In certain embodiments, one camera can be controlled by a first robotic arm and a second camera by another robotic arm. In such a robotic system, one camera may be, for example, a driven camera on a driven arm. The driven arm and the camera on it can be programmed, for example, to track the other camera and maintain a specific distance and / or lens angle.

[0052] In yet another embodiment, the surgical visualization system 100 employs two separate waveform receivers (i.e., cameras / image sensors), d w This may be determined. Referring here to Figure 9, if an important structure 301 or its contents (e.g., a blood vessel or the contents of a blood vessel) can emit a signal 302 by fluoroscopy or the like, its actual position can be triangulated from two separate cameras 320a, 320b at a known position.

[0053] Referring now to Figures 10A and 10B, in another embodiment, the surgical visualization system employs a dithering or moving camera 440, at a distance d wThis may be determined. Camera 440 is robotically controlled such that its three-dimensional coordinates at different positions are known. In various examples, camera 440 can pivot on a cannula or patient interface. For example, if a critical structure 401 or its contents (e.g., blood vessels or contents of a container) can emit a signal, for example by fluoroscopy, its actual position can be triangulated from camera 440 rapidly moving between two or more known positions. In Figure 10A, camera 440 is moved axially along axis A. More specifically, camera 440 is translated along axis A to a position indicated as position 440' by moving in and out with a robotic arm, for example. As camera 440 moves distance d1 and the size of the view relative to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) may correspond to angles θ1 of 6.28 degrees and angles θ2 of 8.19 degrees. Additionally, or alternatively, the camera 440 can rotate or sweep along an arc between different positions. Referring here to Figure 10B, the camera 440 moves axially along axis A and rotates θ3 about axis A. The pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. In Figure 10B, the camera 440 translates and rotates to position 440''. As the camera 440 moves and the edge of the view changes relative to the critical structure 401, the distance to the critical structure 401 can be calculated. In Figure 10B, the distance d2 can be, for example, 9.01 mm, and the angle θ3 can be, for example, 0.9 degrees.

[0054] Figure 5 shows a surgical visualization system 500, which is similar in many respects to the surgical visualization system 100. In various examples, the surgical visualization system 500 may be a further illustration of the surgical visualization system 100. Like the surgical visualization system 100, the surgical visualization system 500 includes a surgical device 502 and an imaging device 520. The imaging device 520 includes a spectral light emitter 523 configured to emit spectral light of multiple wavelengths to acquire spectral images of hidden structures, for example. The imaging device 520 may also include a three-dimensional camera and associated electronic processing circuits in various examples. The surgical visualization system 500 is illustrated, which is used during surgery to identify certain critical structures such as the ureter 501a and blood vessels 501b within an organ 503 (in this example, the uterus) that is not visible on the surface, and to facilitate their avoidance.

[0055] The surgical visualization system 500 uses structured light to measure the emitter-tissue distance d from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503. e The surgical visualization system 500 is configured to determine the emitter-tissue distance d e Based on this, the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 t It is configured to extrapolate. In addition, the surgical visualization system 500 measures the tissue-ureteral distance d from the ureter 501a to the surface 505. A , and the camera-ureteral distance d from the imaging device 520 to the ureter 501a w It is configured to determine the distance d, for example, by spectral imaging and time-of-flight sensors. As described herein with respect to Figure 1, for example, the surgical visualization system 500 determines the distance d by, for example, spectral imaging and time-of-flight sensors. w The tissue-ureteral distance d can be determined based on other distances and / or surface mapping logic described herein. In various cases, the surgical visualization system 500 can determine the tissue-ureteral distance d A (That is, depth) can be determined (for example, by triangulation).

[0056] Referring here to Figure 11, a schematic diagram of a control system 600 for a surgical visualization system, such as surgical visualization system 100, is shown. The control system 600 is a transformation system that identifies important structures, particularly when these structures are obscured by other tissues such as fat, connective tissue, blood, and / or other organs, by integrating tissue identification by spectral signature and tissue positioning by structured light. Such techniques may also be useful in detecting tissue mutations, such as differentiating tumors and / or pathological tissues from healthy tissue within an organ.

[0057] The control system 600 is configured to implement a hyperspectral imaging visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field. The control system 600 includes a conversion logic circuit 648 for converting tissue data into information usable by the surgeon. For example, key structures within the anatomical structure can be identified using wavelength-based variable reflectance relative to shielding materials. Furthermore, the control system 600 combines the identified spectral signatures with structural optical data within the image. For example, the control system 600 can be employed to create a three-dimensional dataset for surgical applications in systems using augmented image overlays. The technology can be employed both during and before surgery, incorporating additional visual information. In various examples, the control system 600 is configured to provide alerts to clinicians when one or more key structures are nearby. Various algorithms can be employed to guide robotic and semi-automated approaches based on proximity to surgical procedures and key structures (one or more).

[0058] A projected light array is used to determine the shape and movement of tissue during surgery. Alternatively, flash lidar may be used for tissue surface mapping.

[0059] The control system 600 is configured to detect one or more critical structures, provide image overlays of these critical structures, and measure the distance to the surface of visible tissue and the distance to embedded / buried critical structures. In other examples, the control system 600 may measure the distance to the surface of visible tissue, or detect one or more critical structures and provide image overlays of these critical structures.

[0060] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 may be, for example, a field-programmable gate array (FPGA) as described herein in relation to Figures 2A to 2C, or another preferred circuit configuration. The spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, and for example, C / C++ code may be used. The video input processor 606 accepts a video input terminal for control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608, which includes, for example, a hyperspectral video output terminal for interface control (metadata) data. The video output processor 608 provides the video output signal to an image overlay controller 610.

[0061] The video input processor 606 is connected to the patient-side camera 612 via a patient isolation circuit 614. As previously mentioned, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include a number of transformers to isolate the patient from other circuits in the system. The camera 612 receives intraoperative images via an optical element 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or any of the image sensor technologies discussed herein in relation to, for example, Figure 2. In one embodiment, the camera 612 outputs an image with a 14-bit / pixel signal. It will also be understood that higher or lower pixel resolutions may be employed without departing from the scope of this disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fusion output signal is provided to the video input processor 606 and the laser pulse control circuit 622.

[0062] The laser pulse control circuit 622 controls the laser light engine 624. The laser light engine 624 emits multiple wavelengths (λ1, λ2, λ3...λ) including near-infrared light (NIR). n The laser light engine 624 outputs light in the following mode. The laser light engine 624 can operate in multiple modes. In one embodiment, the laser light engine 624 can operate in, for example, two modes. In the first mode, for example, the standard operating mode, the laser light engine 624 outputs an illumination signal. In the second mode, for example, the specific mode, the laser light engine 624 outputs RGBG light and NIR light. In various examples, the laser light engine 624 can operate in polarization mode.

[0063] The optical output 626 from the laser light engine 624 illuminates the targeted anatomical structure within the surgical site 627 during surgery. The laser pulse control circuit 622 also controls the laser pulse controller 628 for the laser pattern projector 630, which projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, onto the surgical tissue or organs at the surgical site 627 at a predetermined wavelength (λ2). The camera 612 receives the patterned and reflected light output through the camera optical elements 632. The image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and to the video input module 636 for reading the laser beam pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser beam pattern 631 and outputs a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0065] The first video output signal 640 and the second video output signal 642 include data representing the location of important structures on a three-dimensional surface model provided to the integrated module 643. Combined with the data from the video output processor 608 of the spectral control circuit 602, the integrated module 643 calculates the distance d to the buried important structure. A (Figure 1) can be determined (for example, by the triangulation algorithm 644), and the distance d A This can be provided to the image overlay controller 610 via the video output processor 646. The aforementioned conversion logic may include a conversion logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0066] Preoperative data 650 obtained from CT or MRI scans can be used to align or position specific three-dimensionally deformable tissues in various examples. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610, so that such information can be superimposed on the view from the camera 612 and provided to the video monitor 652. The alignment of preoperative data is further described herein and in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,195, “INTEGRATION OF IMAGING DATA,” filed September 11, 2018, each of which is incorporated herein by reference in its entirety.

[0067] The video monitor 652 can output an integrated / enlarged view from the image overlay controller 610. The clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view showing a three-dimensional rendering of the visible tissue and (B) an enlarged view in which one or more hidden important structures are drawn on top of the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can switch between, for example, one or more hidden important structures and / or distance measurements to the surface of the visible tissue.

[0068] The control system 600 and / or various control circuits thereof can be incorporated into various surgical visualization systems disclosed herein.

[0069] Figure 12 shows a structured (or patterned) light system 700 relating to at least one aspect of the present disclosure. As described herein, structured light, for example in the form of stripes or lines, is projected from a light source and / or projector 706 onto the surface 705 of a targeted anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which may be similar in various respects to an imaging device 120 (Figure 1), may be configured to detect the projected light pattern on the surface 705. As the projected pattern deforms upon impact with the surface 705, the visual system can calculate depth and surface information of the targeted anatomical structure.

[0070] In certain cases, invisible (i.e., undetectable) structured light can be used, which can be utilized without interfering with other computer vision tasks where projected patterns may be disruptive. For example, infrared light repeating two exactly opposite patterns or visible light at a very fast frame rate can be used to prevent interference. Structured light is explained further at en.wikipedia.org / wiki / Structured_light.

[0071] As described above, various surgical visualization systems described herein can be used to visualize various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be prevented from being visualized by EMR in the visible portion of the spectrum. In one embodiment, the surgical visualization system can utilize a spectral imaging system to visualize various types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials in the visualized tissue based on the absorption coefficient of the tissue over various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on a specific combination of constituent materials. For example, Figure 13A is a graph 2300 showing how the absorption coefficients of various biomaterials vary over the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 is the absorption coefficient of the biomaterial (e.g., cm²). -1 The horizontal axis 2304 represents the EMR wavelength (e.g., in μm). This graph 2300 further shows the first line 2310 representing the absorption coefficient of water at various EMR wavelengths, the second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, the third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, the fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, the fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and the sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Since different tissue types have different combinations of constituent materials, the tissue type(s) visualized by the surgical visualization system can be identified and differentiated according to the specific combination of constituent materials detected. Therefore, a spectral imaging system can be configured to emit EMR at many different wavelengths, determine the constituent materials of a tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on a specific detected combination of constituent materials.

[0072] An example of the use of spectral imaging techniques to visualize various tissue types and / or anatomical structures is shown in Figure 13B. In Figure 13B, a spectral emitter 2320 (e.g., spectral light source 150) is used by the imaging system to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissues and / or structures of the surgical site 2325 is received by an image sensor 135 (Figure 2), which can visualize the tissues and / or structures, which may be either visible (e.g., located on the surface of the surgical site 2325) or obscured (e.g., beneath other tissues and / or structures in the surgical site 2325). In this example, the imaging system 142 (Figure 2) can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissues that cannot be identified against known or expected spectral signatures) based on spectral signatures characterized by different absorption properties (e.g., absorption coefficients) of the constituent materials for each of various tissue / structure types. The visualized tissues and structures can be displayed on display screens associated with or connected to the imaging system 142, such as the imaging system display 146 (Figure 2), primary display 2119 (Figure 18), non-sterile display 2109 (Figure 18), hub display 2215 (Figure 19), and device / instrument display 2237 (Figure 19).

[0073] Furthermore, the imaging system 142 can be configured to adjust or update the visualization of the displayed surgical site according to the identified tissue and / or structural type. For example, the imaging system 142 can display a margin 2330a associated with a tumor 2332 visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue to be excised to ensure complete removal of the tumor 2332. The control system 133 (Figure 2) can be configured to control or update the dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 identifies several anomalies 2338 within the FOV. Therefore, the control system 133 can adjust the displayed margin 2330a to match a first updated margin 2330b that has sufficient dimensions to encompass these anomalies 2338. Furthermore, the imaging system 142 identifies artery 2334 that partially overlaps with the initially displayed margin 2330a (shown in the highlighted region 2336 of artery 2334). Therefore, the control system 133 can adjust the displayed margin 2330a to match the second updated margin 2330c, which has sufficient dimensions to encompass the relevant portion of artery 2334.

[0074] Furthermore, tissues and / or structures can be imaged or characterized according to their reflectivity across the EMR wavelength spectrum, in addition to or instead of the absorption characteristics described above with respect to Figures 13A and 13B. For example, Figures 13C to 13E show various graphs of the reflectivity of different types of tissues or structures across various EMR wavelengths. Figure 13C is a graphical representation of an exemplary ureteral signature versus a shielding material. Figure 13D is a graphical representation of an exemplary arterial signature versus a shielding material. Figure 13E is a graphical representation of an exemplary nerve signature versus a shielding material. The plots in Figures 13C to 13E show the reflectivity of specific structures (ureters, arteries, and nerves) as a function of wavelength (in nm) relative to the corresponding reflectivity of fat, lung tissue, and blood at the corresponding wavelengths. These graphs are for illustrative purposes only, and it should be understood that other organizations and / or structures may include corresponding detectable reflectance signatures that enable the identification and visualization of the organization and / or structure.

[0075] In various applications, selective wavelengths for spectral imaging can be identified and utilized based on anticipated important structures and / or obstructions at the surgical site (i.e., “selective spectral” imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized so that information can be acquired in real time or near real time and used intraoperatively. In various applications, wavelengths can be selected by the clinician or by a control circuit based on clinician input. In specific applications, wavelengths can be selected based on machine learning and / or big data accessible to the control circuit, for example, via the cloud.

[0076] The aforementioned example of applying spectral imaging to tissue can be used during surgery to measure the distance between a waveform emitter and critical structures shielded by tissue. In one aspect of this disclosure, a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown, with reference to Figures 14 and 15. In a particular example, the time-of-flight sensor system 1104 can be incorporated into a surgical visualization system 100 (Figure 1). The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted wave 1124 extends from the emitter 1106 to the critical structure 1101, and the received wave 1125 is reflected from the critical structure 1101 to the receiver 1108. The surgical device 1102 is positioned through a trocar 1110 extending into the patient's cavity 1107.

[0077] Waveforms 1124 and 1125 are configured to transmit through the shielding tissue 1103. For example, the wavelengths of waveforms 1124 and 1125 may be wavelengths of the NIR spectrum or SWIR spectrum. In one embodiment, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or a photoacoustic signal can be emitted from the emitter 1106 and can transmit through the tissue 1103 that is obscuring the critical structure 1101. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various examples, waveforms 1124 and 1125 can be selected to target the critical structure 1101 in the tissue 1103 based on the spectral signature of the critical structure 1101, as further described herein. In various examples, the emitter 1106 is configured to supply a binary signal representing on and off, as shown in Figure 15, for example, which can be measured by the receiver 1108.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (Figure 14). The time-of-flight timing of the emitter 1106 and receiver 1108 in Figure 14 is shown in Figure 15. The delay is a function of the distance d, which is given as follows:

[0079]

number

[0080] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to distance d in Figure 14. In various examples, additional emitter / receiver and / or pulse signals from emitter 1106 can be configured to emit an opaque signal. The opaque structure can be configured to determine the distance from the emitter to the surface 1105 of the shielding structure 1103. In various examples, the depth of critical structure 1101 can be determined by the following formula. d A =d w -d t . During the ceremony, d A =This is the depth of important structure 1101, d w = This is the distance from emitter 1106 to critical structure 1101 (d in Figure 14), and, d t = This is the distance from the emitter 1106 (above the distal end of the surgical device 1102) to the surface 1105 of the shielding r tissue 1103.

[0081] In one aspect of this disclosure, with reference to Figure 16, a time-of-flight sensor system 1204 is shown that utilizes waves 1224a, 1224b, 1224c, 1225a, 1225b, and 1225c. In certain examples, the time-of-flight sensor system 1204 can be incorporated into a surgical visualization system 100 (Figure 1). The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is located on a first surgical device 1202a, and the waveform receiver 1208 is located on a second surgical device 1202b. The surgical devices 1202a and 1202b are located through trocars 1210a and 1210b, respectively, which extend within the patient's cavity 1207. The emitted waves 1224a, 1224b, and 1224c extend from the emitter 1206 toward the surgical site, and the received waves 1225a, 1225b, and 1225c are reflected from the receiver 1208 by various structures and / or surfaces at the surgical site.

[0082] Different emitted waves 1224a, 1224b, and 1224c are configured to target different types of material at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first important structure 1201a (e.g., blood vessels), and wave 1224c targets a second important structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, and 1224c may be wavelengths of visible light, NIR, or SWIR spectra. For example, visible light can be reflected by the surface 1205 of tissue 1203, and NIR and / or SWIR waveforms can be configured to transmit through the surface 1205 of tissue 1203. In various embodiments, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from the emitter 1206 as described herein. In various examples, waves 1224b and 1224c can be selected to target key structures 1201a and 1201b within tissue 1203 based on their spectral signatures, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, each of which is incorporated herein by reference in whole.

[0083] The emitted waves 1224a, 1224b, and 1224c can be reflected from the target material (i.e., surface 1205, first critical structure 1201a, and second structure 1201b, respectively). The received waveforms 1225a, 1225b, and 1225c are reflected at the distance d shown in Figure 16. 1a d 2a d 3a d 1b d 2b d 2c Delays may occur due to this.

[0084] In a time-of-flight sensor system 1204 in which the emitter 1206 and receiver 1208 can be positioned individually (for example, on separate surgical devices 1202a, 1202b and / or controlled by a separate robotic arm), various distances d 1a d 2a d 3ad 1b d 2b d 2c This can be calculated from the known positions of the emitter 1206 and receiver 1208. For example, when surgical devices 1202a and 1202b are robotically controlled, their positions may be known. With knowledge of the positions of the emitter 1206 and receiver 1208, as well as the time of the photon stream targeting a specific tissue and the information of that specific response received by receiver 1208, the distance d can be calculated. 1a d 2a d 3a d 1b d 2b d 2c The distance can be determined. In one embodiment, the distance to the shielded critical structures 1201a, 1201b can be determined by triangulation using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0085] Referring further to Figure 16, in various examples, the view provided to the clinician allows the receiver 1208 to be rotated in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b, such that the center of mass of the target structure in the resulting image remains constant. Such orientations can quickly communicate one or more relevant distances and / or viewpoints to the important structure. For example, as shown in Figure 16, the surgical site is displayed from a viewpoint where the important structure 1201a is perpendicular to the view plane (i.e., blood vessels are facing inward and outward from the page). In various examples, such orientations may be the default setting. However, the view can be rotated or adjusted by the clinician. In certain examples, the clinician can switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 can be mounted on a trocar or cannula, such as a trocar 1210b, through which the surgical device 1202b is positioned. In other examples, the receiver 1208 can be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 can be mounted on a movable arm separate from the robot controlling the surgical device 1202a, or on an operating room (OR) table that can be aligned to the robot coordinate plane during surgery. In such examples, the positions of the emitter 1206 and the receiver 1208 can be aligned to the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] The combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS) known as TOF-NIRS, which enables the measurement of time-resolved profiles of NIR light with nanosecond resolution, can be found in the Journal of the American Society for Horticultural Science, May 2013, vol. 138, no. 3, pp. 225-228, titled "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT," which is incorporated herein by reference in its entirety and can be viewed at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, time-of-flight spectral waveforms are configured to determine the depth of critical structures and / or the proximity of surgical devices to critical structures. Furthermore, various surgical visualization systems disclosed herein include surface mapping logic configured to produce a three-dimensional rendering of the surface of visible tissue. In such examples, even if critical structures are obscured by visible tissue, the clinician can recognize the proximity (or lack thereof) of surgical devices to critical structures. In one example, a topography of the surgical site is provided on a monitor by the surface mapping logic. If critical structures are close to the tissue surface, spectral imaging can communicate the location of the critical structures to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other examples, spectral imaging can detect structures 10 or 20 mm below the tissue surface. Based on known limits of the spectral imaging system, the system is configured to communicate that critical structures are out of range if they are not simply detected by the spectral imaging system. Thus, the clinician can continue to move the surgical device and / or manipulate the tissue. When a critical structure moves within the range of a spectral imaging system, the system can identify the structure and therefore communicate that the structure is within range. In such cases, an alert can be provided when the structure is first identified and / or moves further within a predetermined proximity area. In such cases, even if the critical structure is not identified by a spectral imaging system with known boundaries / ranges, the clinician can be provided with proximity information (i.e., not in proximity).

[0089] Various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures(s) during surgery and to warn clinicians before inadvertently damaging critical structures(s) through inadvertent incisions and / or transverse incisions. In various embodiments, the surgical visualization system is configured to identify one or more of the following important structures, e.g., ureters, intestines, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], process facial nerve, vagus nerve, and their branches), blood vessels (including arteries and veins of the lungs and lobes, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including the middle colic artery, right colic artery, and ileocolic artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal iliac vessels (lower abdomen), short gastric arteries, uterine arteries, median sacral vessels, and lymph nodes. Furthermore, the surgical visualization system is configured to indicate the proximity of one or more surgical devices to one or more critical structures, and / or to alert the clinician when one or more surgical devices approach critical structures.

[0090] Various aspects of this disclosure provide identification of critical structures during surgery (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, enabling visualization of critical structures below the surface of tissue, such as 1.0–1.5 cm below the tissue surface. In other examples, surgical visualization systems may identify structures less than 1.0 cm or more than 1.5 cm below the tissue surface. For example, even a surgical visualization system that can only identify structures within 0.2 mm of the surface may be useful in cases where structures are otherwise invisible due to depth. In various aspects, surgical visualization systems can extend the clinician's view by, for example, virtually displaying critical structures as a visible white light image overlay on the surface of visible tissue. Surgical visualization systems can provide real-time three-dimensional spatial tracking of the distal tip of a surgical instrument and can provide proximity alerts when the distal tip of a surgical instrument moves within a specific range of a critical structure, such as within 1.0 cm of the critical structure.

[0091] Various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. An incision may be considered "too close" to a critical structure based on temperature (i.e., too high a temperature near the critical structure, which could risk damaging / heating / melting the critical structure) and / or tension (i.e., too high a tension near the critical structure, which could risk damaging / tearing / pulling the critical structure). Such surgical visualization systems can facilitate perivascular incisions, for example, when skeletonizing perivascular tissue before ligation. In various examples, an infrared camera can be used to read the heat of the surgical site and provide a warning to the clinician based on the detected heat and the distance from the instrument to the structure. For example, if the instrument temperature exceeds a predetermined threshold (e.g., 120°F), an alert can be provided to the clinician at a first distance (e.g., 10 mm), and if the instrument temperature is below the predetermined threshold, an alert can be provided to the clinician at a second distance (e.g., 5 mm). Default thresholds and / or warning distances may be default settings and / or programmable by the clinician. Additionally, or alternatively, proximity alerts may be linked to thermal measurements performed on the instrument itself, such as unipolar or bipolar cutting instruments, or thermocouples measuring heat within the distal jaws of a vascular sealer.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity to critical structures to enable clinicians to confidently and rapidly but safely perform incisions based on caution criteria and / or device safety data. These systems can function in real time during surgical procedures with minimal ionizing radiation risk to the patient or clinician, and in various cases, there is no ionizing radiation risk to the patient or clinician. Conversely, in fluoroscopy procedures, the patient and / or clinician(s) may be exposed to ionizing radiation, for example, via an X-ray beam used to visualize anatomical structures in real time.

[0093] Various surgical visualization systems disclosed herein can be configured to detect and identify one or more desired types of critical structures in the forward path of a surgical device, for example, when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more types of critical structures in the periphery of the surgical device and / or in multiple planes / dimensions.

[0094] The various surgical visualization systems disclosed herein are easily operable and / or interpretable. Furthermore, the various surgical visualization systems may incorporate “override” functions that allow clinicians to override default settings and / or operations. For example, clinicians may selectively turn off alerts from the surgical visualization system and / or approach critical structures more closely than suggested by the surgical visualization system when the risk to critical structures is lower than the risk of avoiding that area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue may be greater than the risk of damaging the critical structure).

[0095] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on the workflow. In other words, the implementation of the surgical visualization system does not need to change the way the surgical procedure is performed. Furthermore, the surgical visualization system may be more economical compared to the costs associated with accidental transverse incisions. The data shows a reduction in accidental damage to critical structures, which can lead to increased reimbursement.

[0096] The various surgical visualization systems disclosed herein can operate in real time, near real time, and well in advance to enable clinicians to predict critical structures (one or more). For example, a surgical visualization system can provide sufficient time for "slowing down, evaluating, and avoiding" to maximize the efficiency of surgical procedures.

[0097] The various surgical visualization systems disclosed herein do not require contrast agents or dyes to be injected into the tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other examples, contrast agents may be easier to inject into the appropriate layer(s) of the tissue than other visualization systems. The time between contrast agent injection and visualization of important structures may be, for example, less than two hours.

[0098] The various surgical visualization systems disclosed herein may be linked to clinical data and / or instrument data. For example, the data may provide a boundary indicating how close an energy-supplied surgical device (or other device that could potentially damage) is to tissue that the surgeon does not want to damage. Any data module interfacing with the surgical visualization systems disclosed herein may be provided integrated with or separately from a robot to enable use with a standalone surgical device in open or laparoscopic procedures, for example. The surgical visualization systems may be compatible with robotic surgical systems in various examples. For example, the visualization images / information may be displayed within the robot console.

[0099] In various cases, clinicians may be unaware of the location of critical structures relative to surgical instruments. For example, if critical structures are embedded in tissue, it may be impossible for clinicians to locate them. In certain cases, clinicians may want to keep surgical devices outside the range surrounding critical structures and / or away from visible tissue covering hidden critical structures. If the location of hidden critical structures is unknown, clinicians risk getting too close to them, potentially resulting in inadvertent trauma and / or incision of the critical structures, and / or applying excessive energy, heat, and / or tension near them. Alternatively, clinicians may remain too far away from suspected critical structures, risking affecting tissue in less desirable locations while attempting to avoid them.

[0100] A surgical visualization system is provided that presents tracking of a surgical device to one or more critical structures. For example, the surgical visualization system can track the proximity of a surgical device to a critical structure. Such tracking can be performed in real time and / or near real time during surgery. In various examples, the tracking data can be provided to the clinician via the display screen (e.g., monitor) of the imaging system.

[0101] In one aspect of this disclosure, the surgical visualization system comprises a surgical device having an emitter configured to emit a structured light pattern onto a visible surface; an imaging system having a camera configured to detect an implanted structure and the structured light pattern on a visible surface; and a control circuit communicating with the camera and the imaging system, wherein the control circuit is configured to determine the distance from the surgical device to the implanted structure and to provide the imaging system with a signal indicating the distance. For example, the distance can be determined based on a three-dimensional view of the illuminated structure provided by images from multiple lenses of the camera (e.g., left lens and right lens) by calculating the distance from the camera to a critical structure that glows by fluorescence fluoroscopy. The distance from the surgical device to the critical structure can be determined, for example, by triangulation based on known positions of the surgical device and the camera. Alternative means for determining the distance to the implanted critical structure are further described herein. For example, an NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue overlapping / covering the implanted critical structure. For example, a surgical visualization system can identify hidden critical structures and extend the view of these hidden structures by outlining them on the visible tissue, such as lines on the surface of the visible tissue. The surgical visualization system can further determine the distance to these extended lines on the visible tissue.

[0102] As provided by the various surgical visualization systems disclosed herein, clinicians can make more informed decisions about the placement of surgical devices relative to hidden critical structures by providing clinicians with up-to-date information on the proximity of surgical devices to hidden critical structures and / or visible structures. For example, clinicians can view the distance between surgical devices and critical structures in real time / during surgery, and in certain cases, the imaging system can provide alerts and / or warnings when the surgical device is moving within a predetermined proximity and / or area of ​​the critical structure. In certain cases, alerts and / or warnings may be provided when the trajectory of the surgical device indicates a potential collision with a “no-fly” zone near the critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm or more of the critical structure). In such cases, clinicians can maintain momentum throughout the surgery without requiring clinician monitoring of the suspected location of the critical structure and the proximity of the surgical device to this location. As a result, certain surgical procedures can be performed more quickly with fewer pauses / interruptions and / or improved accuracy and / or certainty. In one embodiment, a surgical visualization system can be used to detect tissue mutations, such as tissue mutations within organs, in order to differentiate tumor / cancer / pathological tissue from healthy tissue. Such a surgical visualization system can maximize the removal of pathological tissue while minimizing the removal of healthy tissue.

[0103] Surgical hub system The various visualization systems, i.e., imaging systems, described herein are illustrated in relation to Figures 17-19 and can be incorporated into surgical hub systems, as further detailed below.

[0104] Referring to Figure 17, the computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include the remote server 2113. In one example, as shown in Figure 17, the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112, which are configured to communicate with each other and / or with the hub 2106. In some embodiments, the surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers of 1 or more.

[0105] Figure 18 shows an example of a surgical system 2102 used to perform surgery on a patient lying on an operating table 2114 in an operating room 2116. A robotic system 2110 is used as part of the surgical system 2102 in the surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (surgical robot), and a surgical robot hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side cart 2120 can manipulate at least one detachably connected surgical instrument 2117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The robotic hub 2122 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 2118.

[0106] Other types of robotic systems can be readily adapted for use with surgical system 2102. Various examples of robotic system surgical instruments suitable for use with this disclosure are described in various U.S. patent applications incorporated herein by reference.

[0107] Various examples of cloud-based analytics performed by Cloud 2104 and suitable for use with this disclosure are described in various U.S. patent applications incorporated herein by reference.

[0108] In various embodiments, the imaging device 2124 includes at least one image sensor and one or more optical components. Preferred image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0109] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light or simply light. The typical human eye responds to wavelengths in air from approximately 380 nm to approximately 750 nm.

[0111] The invisible spectrum (i.e., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0112] In various embodiments, the imaging device 2124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0113] In one embodiment, the imaging device employs multispectral monitoring to distinguish topography from underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. Uses of multispectral imaging are described in various U.S. patent applications incorporated herein by reference. Multispectral monitoring can be a useful tool for repositioning the surgical field after the completion of a surgical task to perform one or more of the tests described above on the treated tissue.

[0114] It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating area,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and equipment. Part of the above sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 2124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or it may be considered the area immediately surrounding a patient ready for surgery. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and restraints within that area. In various embodiments, the visualization system 2108 includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 18. In one embodiment, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 2108 are described in various U.S. patent applications incorporated herein by reference.

[0115] As shown in Figure 18, the primary display 2119 is positioned in the sterile field so that it is visible to the operator on the operating table 2114. In addition, a visualization tower 21121 is positioned outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109, facing opposite directions from each other. The visualization system 2108, guided by the hub 2106, is configured to utilize displays 2107, 2109, and 2119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 2106 can cause the visualization system 2108 to display snapshots of the surgical site recorded by the imaging device 2124 on the non-sterile displays 2107 or 2109 while maintaining live video of the surgical site on the primary display 2119. The snapshots on the non-sterile displays 2107 or 2109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0116] In one embodiment, the hub 2106 is also configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 21121 to a primary display 2119 in the sterile field, which can then be viewed by a sterile operator on the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 2107 or 2109, which can then be sent to the primary display 2119 by the hub 2106.

[0117] Referring to Figure 18, surgical instrument 2112 is used in a surgical procedure as part of surgical system 2102. Hub 2106 is also configured to adjust the information flow to match the display of surgical instrument 2112, as described in various U.S. patent applications incorporated herein by reference. Diagnostic input or feedback entered by a non-sterile operator in visualization tower 21121 can be sent by Hub 2106 to surgical instrument display 2115 in the sterile field, which can then be viewed by the operator of surgical instrument 2112. Illustrative surgical instruments suitable for use with surgical system 2102 are described in various U.S. patent applications incorporated herein by reference.

[0118] Figure 19 shows a computer-implemented interactive surgical system 2200. The computer-implemented interactive surgical system 2200 is similar in many respects to the computer-implemented interactive surgical system 2100. Each surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204, which may include a remote server 2213. In one embodiment, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to multiple surgical field devices, such as intelligent surgical instruments, robots, and other computerized devices in the operating room. The surgical hub 2236 includes a communication interface for connecting the surgical hub 2236 to the cloud 2204 and / or the remote server 2213 in a communicative manner. As shown in the embodiment of Figure 19, the surgical hub 2236 is connected to an imaging module 2238 connected to an endoscope 2239, a generator module 2240 connected to an energy device 2421, a smoke exhaust module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235 optionally connected to a display 2237, and a non-contact sensor module 2242. The surgical field devices are connected to cloud computing resources and data storage via the surgical hub 2236. A robot hub 2222 may also be connected to the surgical hub 2236 and cloud computing resources. As described herein, among other things, the device / instrument 2235 and the visualization system 2209 may be connected to the surgical hub 2236 via wired or wireless communication standards or protocols. The surgical hub 2236 can be connected to a hub display 2215 (e.g., a monitor, screen) to display and overlay images received from imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display may also display data received from devices connected to a modular control tower, along with the images and overlaid images.

[0119] Situational awareness Various visualization systems, or embodiments thereof, described herein can be used as part of a situational awareness system that can be embodied or executed by the surgical hubs 2106, 2236 (Figures 17-19). Specifically, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and actions), tissues, structures, users, and other things located in the surgical field or operating room can provide contextual data that can be used by the situational awareness system to infer the type of surgery or process being performed, the type of tissue(s) and / or structure(s) being manipulated by the surgeon, etc. This contextual data can then be used by the situational awareness system to enable the provision of alerts to the user, suggestions for subsequent processes or actions the user should take, preparation in anticipation of the use of surgical devices (e.g., activating an electrosurgical generator in anticipation of the use of an electrosurgical instrument in a post-surgical process), and intelligent control of surgical instruments (e.g., customizing the operating parameters of surgical instruments based on each patient's specific health profile).

[0120] An "intelligent" device that includes a control algorithm that responds to detected data may be an improvement over a "data-dumb" device that operates without considering detected data. However, some detected data, when considered in isolation, may be incomplete or inconclusive without the context of the type of surgery being performed or the type of tissue being operated on. Without knowledge of the surgical context (e.g., the type of tissue being operated on or the type of procedure being performed), a control algorithm may inaccurately or suboptimally control a modular device with certain detected data that lacks context. Modular devices can include visualization system devices (e.g., cameras or display screens), surgical instruments (e.g., ultrasonic surgical instruments, electrosurgical instruments, or surgical staplers), and any surgical devices controllable by a context-aware system, such as other surgical devices (e.g., smoke exhausters). For example, the optimal form of a control algorithm for controlling a surgical instrument in response to specific detected parameters may vary depending on the specific type of tissue being operated on. This is due to the fact that different types of tissue have different properties (e.g., resistance to tearing) and therefore respond differently to actions taken by the surgical instrument. Therefore, even when the same measurement is detected for a particular parameter, it may be desirable for surgical instruments to take different actions. As one specific example, the optimal way in which surgical stapling and cutting instruments are controlled in response to an instrument detecting an unexpectedly high force to close its end effector depends on whether the tissue type is susceptible to tearing or resistant to it. For tear-sensitive tissues, such as lung tissue, the instrument's control algorithm optimally slows down the motor in response to an unexpectedly high force to close in order to avoid tearing the tissue. For tear-resistant tissues, such as stomach tissue, the instrument's control algorithm optimally accelerates the motor in response to an unexpectedly high force to close in order to ensure that the end effector is properly clamped to the tissue.If it's unclear whether lung tissue or stomach tissue is being clamped, the control algorithm may make suboptimal decisions.

[0121] One solution utilizes a surgical hub, which includes a system configured to derive information about the surgical procedure being performed based on data received from various data sources, and then appropriately control paired modular devices. In other words, the surgical hub is configured to infer information about the surgical procedure from the received data, and then control the modular devices paired with the surgical hub based on the inferred context about the surgical procedure. Figure 20 shows a diagram of a context-aware surgical system 2400 relating to at least one aspect of the present disclosure. In some examples, the data source 2426 may include, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database containing patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0122] The surgical hub 2404 may be similar in many respects to the hub 106, for example, it may be configured to derive contextual information about a surgical procedure from data based on a specific combination(single or multiple) of received data, or a specific order in which data is received from the data source 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function relating to some aspects of the surgical hub 2404 for deriving or inferring information about a surgical procedure from received data is sometimes referred to as “situational awareness.” In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to a surgical procedure from received data.

[0123] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data source 2426 in various different ways. In one example, the situational awareness system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular device 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another example, the situational awareness system may include a lookup table that stores pre-characterized contextual information about the surgical procedure, associated with one or more inputs (or ranges of inputs) that correspond to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situational awareness system to control the modular device 2402. In one example, contextual information received by the situation awareness system of the surgical hub 2404 is associated with a specific control adjustment, or a set of control adjustments, of one or more modular devices 2402. In another example, the situation awareness system includes a further machine learning system, lookup table, or other such system that, given contextual information as input, generates or retrieves one or more control adjustments of one or more modular devices 2402.

[0124] The surgical hub 2404, which incorporates a situational awareness system, brings many advantages to the surgical system 2400. One advantage is improved interpretation of detected and collected data, which improves processing accuracy during the surgical procedure and / or data utilization. Returning to the previous example, the situational awareness surgical hub 2404 can determine what type of tissue is being operated on, so if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 2404 can correctly accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0125] In another embodiment, the type of tissue being operated on may affect the adjustments made to the compression rate and load threshold of surgical stapling and cutting instruments for measuring specific interstitial gaps. The situation-aware surgical hub 2404 can infer whether the surgery being performed is a thoracic or abdominal surgery, thereby allowing the surgical hub 2404 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is the lung (in the case of thoracic surgery) or the stomach (in the case of abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0126] In yet another embodiment, the type of body cavity being operated on during the air insufflation procedure may affect the function of the smoke exhauster. The situation-aware surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing air insufflation) and determine the type of procedure. Generally, since certain types of procedures are performed in specific body cavities, the surgical hub 2404 can appropriately control the motor speed of the smoke exhauster to match the body cavity being operated on. Thus, the situation-aware surgical hub 2404 can provide a consistent amount of smoke exhaust for both thoracic and abdominal surgeries.

[0127] As another example, the type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situation-aware surgical hub 2404 can determine whether the surgical procedure is an arthroscopy. The surgical hub 2404 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situation-aware surgical hub 2404 can determine what type of surgical procedure is being performed and then customize the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument according to the expected tissue shape for the surgical procedure. Furthermore, the situation-aware surgical hub 2404 can be configured to adjust the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument not simply per procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 2404 can determine which steps of a surgical procedure are being performed or are continuing, and then update the control algorithms for the generator and / or ultrasonic or RF electrosurgical instruments to set the energy level to an appropriate value for the expected tissue type according to the steps of the surgical procedure.

[0128] As yet another example, the surgical hub 2404 may also derive data from additional data sources 2426 to improve conclusions drawn from one data source 2426. The contextually aware surgical hub 2404 may enhance data received from the modular device 2402 with contextual information constructed from other data sources 2426 regarding the surgical procedure. For example, the contextually aware surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 2404 may be further configured to make a determination regarding the integrity of the staple line or tissue weld by comparing physiological measurements (e.g., blood pressure detected by a BP monitor communicably connected to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 2404). In other words, the context-aware system of the surgical hub 2404 can provide additional context when analyzing visualization data by considering physiological measurement data. This additional context can be useful when the visualization data itself may not be definitive or may be incomplete.

[0129] Another advantage is the proactive and automatic control of paired modular devices 2402 according to specific steps of the surgical procedure being performed, in order to reduce the number of times medical personnel are required to interact with or control the surgical system 2400 during the course of the surgical procedure. For example, the situation-aware surgical hub 2404 can proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that the use of the instrument is required in a subsequent step of the procedure. By proactively activating the energy source, the instrument can be ready for use as soon as the preceding step of the procedure is completed.

[0130] In another embodiment, the situational awareness surgical hub 2404 can determine whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display, according to the shape(s) of the surgical site that the surgeon is expected to need to see. The surgical hub 2404 can then appropriately and proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0131] As yet another example, the situation-aware surgical hub 2404 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons of data are required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0132] Another advantage is the ability to check for errors during surgical setup or during the surgical procedure itself. For example, the situation-aware surgical hub 2404 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to compare, for example, a list of items for the procedure scanned with a suitable scanner, and / or a list of devices to be paired with the surgical hub 2404, with a given recommended or predicted catalog of items and / or devices for the surgical procedure. If discontinuities exist between the lists, the surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical items are missing. For example, the surgical hub 2404 may be configured to determine the relative distance or relative position of the modular device 2402 and the patient monitoring device 2424, for instance, by proximity sensors. The surgical hub 2404 can compare the relative positions of the devices to a recommended or predicted layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 2404 may be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0133] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other healthcare professional) is making an error or deviating from a set of actions expected during the course of surgery. For example, the surgical hub 2404 may be configured to determine the type of surgery being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the course of surgery with the steps or instruments expected for the type of surgery that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 may be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgery.

[0134] Overall, the context-aware system for the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (e.g., to suit different tissue types) and by validating actions during surgery. The context-aware system also improves surgeon efficiency when performing surgery by automatically suggesting the next steps, providing data, and adjusting the in-surgery displays and other modular devices 2402 according to the specific context of the procedure.

[0135] Referring here to Figure 21, a timeline 2500 is shown illustrating the contextual awareness of a hub, such as surgical hub 106 or 206 (Figures 1-11). Timeline 2500 shows an exemplary surgical procedure and the contextual information that surgical hubs 106, 206 can derive from data received from data sources at each stage of the surgical procedure. Timeline 2500 shows the typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy, starting with setting up the operating room and ending with transferring the patient to the postoperative recovery room.

[0136] The context-aware surgical hubs 106 and 206 receive data from data sources, including data generated each time a medical professional uses a modular device paired with the surgical hubs 106 and 206 throughout the course of a surgical procedure. The surgical hubs 106 and 206 receive this data from paired modular devices and other data sources, and as new data is received, such as which step of the procedure is being performed at any given time, they can continuously derive inferences (i.e., contextual information) about the ongoing procedure. The context-aware system of the surgical hubs 106 and 206 can, for example, record data about the procedure to generate a report, verify the steps being taken by the medical professional, provide data or prompts that may be relevant to a particular surgical step (e.g., via a display screen), adjust modular devices based on context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic or RF electrosurgical instrument), and perform any other of the above actions.

[0137] As the first step 2502 in this exemplary procedure, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hubs 106, 206 determine that the procedure to be performed is a thoracic surgery.

[0138] In step 2504, personnel scan the medical supplies that have arrived for the procedure. Surgical hubs 106 and 206 cross-reference the scanned supplies with a list of supplies used in various types of procedures to confirm that the combination of supplies matches a chest procedure. Furthermore, surgical hubs 106 and 206 can also determine that the procedure is not a wedge procedure (for example, if the arriving supplies do not contain specific supplies required for a chest wedge procedure or are otherwise not compatible with a chest wedge procedure).

[0139] In the third step 2506, the medical professional scans the patient band via a scanner that is communicatively connected to the surgical hubs 106, 206. The surgical hubs 106, 206 can then verify the patient's identification information based on the scanned data.

[0140] In step 4, 2508, medical staff turn on the auxiliary devices. The auxiliary devices used may vary depending on the type of surgery and the techniques used by the surgeon, but in this exemplary case, they include a fume exhauster, an air insulator, and a medical imaging device. Once the auxiliary devices are activated, the modular devices can automatically pair with surgical hubs 106, 206 located within a specific vicinity of the modular devices as part of their initialization process. The surgical hubs 106, 206 can then derive contextual information about the surgery by detecting the type of modular device paired during this pre-operative or initialization phase. In this particular example, the surgical hubs 106, 206 determine that the surgery is a VATS surgery based on this specific combination of paired modular devices. Based on the data from the patient's EMR, the list of medical supplies to be used in the procedure, and the type of modular device connected to the hub, the surgical hubs 106, 206 can generally infer the specific procedure to be performed by the surgical team. When the surgical hubs 106 and 206 recognize which particular procedure is being performed, they then read the steps of that surgery from memory or the cloud, and then cross-reference the data they continue to receive from connected data sources (e.g., modular devices and patient monitoring devices) to infer which steps of the surgical procedure the surgical team is performing.

[0141] In step 5, 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hubs 106 and 206. Once the surgical hubs 106 and 206 begin receiving data from the patient monitoring devices, they confirm that the patient is in the operating room.

[0142] In step 6, 2512, the medical personnel administer anesthesia to the patient. The surgical hubs 106, 206 can infer that the patient is under anesthesia based on data from modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once step 6, 2512 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion commences.

[0143] Step 7, 2514, collapses the lung of the patient being operated on (while simultaneously switching ventilation to the contralateral lung). Surgical hubs 106, 206 can infer, for example, that the patient's lung has collapsed from ventilator data. Since the surgical hubs 106, 206 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), they can infer that the surgical portion of the procedure has started, thereby determining that lung collapse is the first surgical step in this particular procedure.

[0144] In step 8, 2516, a medical imaging device (e.g., a scope) is inserted, and video recording from the medical imaging device is initiated. The surgical hubs 106 and 206 receive medical imaging device data (i.e., video data or image data) through their connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hubs 106 and 206 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hubs 106 and 206 can determine that the specific procedure being performed is a segmentectomy and not a lobectomy (note that, based on the data received in step 2, 2504 of the surgery, a wedge procedure has already been determined to be unlikely by the surgical hubs 106 and 206). Using data from the medical imaging device 124 (Figure 2), contextual information about the type of procedure being performed can be determined in several different ways, including determining the angle at which the medical imaging device is directed with respect to the visualization of the patient's anatomical structure, monitoring the number of medical imaging devices being used (i.e., activated and paired with the surgical hubs 106, 206), and monitoring the type of visualization device being used. For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. The context recognition system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing VATS segmentectomy utilizes an infrared light source (which can be connected to a surgical hub for communication as part of the visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy.By tracking any or all of this data from the medical imaging device, the surgical hubs 106, 206 can determine the specific type of surgical procedure being performed and / or the techniques being used in that specific type of surgical procedure.

[0145] In step 9, 2518, the surgical team begins the incision phase of the procedure. The surgical hubs 106, 206 receive data from the RF or ultrasound generator indicating that an energy device is being emitted, allowing them to infer that the surgeon is in the process of incising and manipulating the patient's lung. The surgical hubs 106, 206 cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy device being emitted at this point in the process (i.e., after the completion of the procedure described above) corresponds to the incision phase. In certain examples, the energy device may be an energy device mounted on the robotic arm of a robotic surgical system.

[0146] In step 10, 2520, the surgical team proceeds to the ligation step of the procedure. The surgical hubs 106 and 206 receive data from the surgical stapling and cutting instruments indicating that the instruments are being fired, so they can infer that the surgeon is currently ligating arteries and veins. As in the previous step, the surgical hubs 106 and 206 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the read steps of the process. In a particular example, the surgical instruments may be surgical instruments mounted on the robotic arms of a robotic surgical system.

[0147] In step 11, 2522, the segmental resection of the procedure is performed. The surgical hubs 106, 206 can infer that the surgeon is transversely incising the parenchyma based on data from the surgical stapling and cutting instruments, including data from their cartridges. The data from the cartridges may correspond, for example, to the size or type of staples fired by the instruments. Since different types of staples are used for different types of tissue, the data from the cartridges may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples fired is used for parenchyma (or other similar tissue types), thereby enabling the surgical hubs 106, 206 to infer that the segmental resection of the procedure is being performed.

[0148] Next, in step 12, 2524, the nodule incision step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hubs 106, 206 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby enabling the surgical hubs 106, 206 to make the above inference. Note that the surgeon may periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Furthermore, in certain examples, robotic tools may be used in one or more steps of the surgical procedure and / or handheld surgical instruments may be used in one or more steps of the surgical procedure. The surgeon(s) may, for example, alternately use robotic tools and handheld surgical instruments, and / or use the devices simultaneously. Once step 12 (2524) is completed, the incision is closed and the postoperative portion of the procedure begins.

[0149] In step 13, 2526, the patient is awakened from anesthesia. The surgical hubs 106, 206 can infer that the patient is waking up from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate beginning to increase).

[0150] Finally, step 14, 2528, is for the medical professional to remove various patient monitoring devices from the patient. Thus, the surgical hubs 2106, 2236 can infer that the patient has been transferred to the recovery room when the hubs lose EKG data, BP data, and other data from the patient monitoring devices. As can be seen from this exemplary surgical description, the surgical hubs 2106, 2236 can determine or infer when each step of a given surgical procedure is being performed, based on the data received from various data sources that are communicably connected to the surgical hubs 2106, 2236.

[0151] Situational awareness is further described in various U.S. patent applications incorporated herein by reference, which are incorporated herein by reference in their entirety. In certain examples, the operation of a robotic surgical system, including, for example, various robotic surgical systems disclosed herein, can be controlled by hubs 2106, 2236 based on its situational awareness and / or feedback from its components, and / or information from cloud 2104 (Figure 17).

[0152] Figure 22 is a logical flowchart of a process 4000 that shows a control program or logical configuration for correlating visualization data and instrument data, relating to at least one aspect of the present disclosure. Process 4000 is generally performed during a surgical procedure and includes, 4001, receiving or deriving a first dataset, i.e., visualization data, that shows the visual aspects of the surgical instrument relative to the surgical field; 4002, receiving or deriving a second dataset, i.e., instrument data, that shows the functional aspects of the surgical instrument; and 4003, correlating the first dataset and the second dataset.

[0153] In at least one example, correlating visualization data with instrument data is implemented by constructing a composite dataset from the visualization data and instrument data. Process 4000 may further include comparing the composite dataset with another composite dataset that can be received from an external source and / or derived from a previously collected composite dataset. In at least one example, process 4000 includes displaying a comparison of the two composite datasets, as described in more detail below.

[0154] The visualization data of process 4000 can show the visual aspect of the end effector of a surgical instrument relative to the tissue in the surgical field. Additionally or alternatively, the visualization data can show the visual aspect of the tissue being treated by the end effector of the surgical instrument. In at least one example, the visualization data represents the position of one or more of the end effector or its components relative to the tissue in the surgical field. Additionally or alternatively, the visualization data may represent the movement of one or more of the end effector or its components relative to the tissue in the surgical field. In at least one example, the visualization data represents one or more changes in the shape, dimensions, and / or color of the tissue being treated by the end effector of the surgical instrument.

[0155] In various embodiments, visualization data is derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). Visualization data can be derived from various measurements, readings, and / or any other preferred parameters monitored and / or captured by the surgical visualization system, as will be described in more detail in relation to Figures 1 to 18. In various examples, visualization data shows one or more visual aspects of tissue in the surgical field, and / or one or more visual aspects of surgical instruments relative to tissue in the surgical field. In specific examples, visualization data represents or identifies the position and / or movement of the end effector of a surgical instrument relative to a critical structure in the surgical field (e.g., critical structure 101 in Figure 1). In specific examples, visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data calculated by surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, and 141.

[0156] In at least one example, visualization data is derived from tissue identification and geometric surface mapping performed by the visualization system 100 in combination with the distance sensor system 104, as described in more detail in relation to Figure 1. In at least one example, visualization data is derived from measurements, readings, or any other sensor data acquired by the imaging device 120. As described in relation to Figure 1, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on molecular responses to various wavelengths.

[0157] Additionally, or alternatively, visualization data can be derived from measurements, readings, or any suitable sensor data acquired by any suitable imaging device, including, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In at least one example, as described in more detail in relation to Figures 3-4 and 13-16, visualization data is derived from a visualization system 160, which includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. In yet another example, visualization data is derived from a visualization system including a three-dimensional (3D) camera and associated electronic processing circuits, such as a visualization system 500. In yet another example, visualization data is derived from a structured (or patterned) light system 700, which will be described in more detail in relation to Figure 12. The aforementioned examples can be used individually or in combination to derive visualization data for process 4000.

[0158] The instrument data of process 4000 can represent one or more actions of one or more internal components of a surgical instrument. In at least one example, the instrument data represents one or more action parameters of one or more internal components of a surgical instrument. The instrument data may represent one or more positions and / or one or more movements of one or more internal components of a surgical instrument. In at least one example, an internal component is a cutting member configured to cut tissue during the firing sequence of the surgical instrument. Additionally, or alternatively, an internal component may include one or more staples configured to be fired into tissue during the firing sequence of the surgical instrument.

[0159] In at least one example, instrument data represents one or more actions of one or more components of one or more drive assemblies of a surgical instrument, such as a joint motion drive assembly, a closing drive assembly, a rotational drive assembly, and / or a firing drive assembly. In at least one example, instrument data sets represent one or more actions of one or more drive members of a surgical instrument, such as a joint motion drive member, a closing drive member, a rotational drive member, and / or a firing drive member.

[0160] Figure 23 is a schematic diagram of an exemplary surgical instrument 4600 for use with process 4000, which is similar in many respects to other surgical instruments or surgical devices described in this disclosure, such as surgical instrument 2112. For brevity, various embodiments of process 4000 are described only in this disclosure using handheld surgical instruments. However, this is not limiting. Such embodiments of process 4000 can equally be implemented with robotic surgical instruments, such as surgical instrument 2117.

[0161] The surgical instrument 4600 includes multiple motors that can be activated to perform various functions. The multiple motors of the surgical instrument 4600 can be activated to cause firing, closing, and / or articulating motions in the end effector. The firing, closing, and / or articulating motions can be transmitted to the end effector of the surgical instrument 4600, for example, via a shaft assembly. However, in other examples, the surgical instrument for use with process 4000 may be configured to manually perform one or more of the firing, closing, and articulating motions. In at least one example, the surgical instrument 4600 includes an end effector that treats tissue by deploying staples into the tissue. In another example, the surgical instrument 4600 includes an end effector that treats tissue by applying therapeutic energy to the tissue.

[0162] In a particular example, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 can be operably coupled to a firing motor drive assembly 4604, which can be configured to transmit the firing motion generated by the firing motor 4602 to an end effector in order to move a firing member in the form of an I-beam, which may include a cutting member. In a particular example, the firing motion generated by the firing motor 4602 can deploy a staple from a staple cartridge into tissue captured by the end effector, and optionally advance the cutting member of the I-beam to cut the captured tissue, for example.

[0163] In certain examples, the surgical instrument or surgical device may include a closure motor 4603. The closure motor 4603 can be operably coupled to a closure motor drive assembly 4605, which can be configured to transmit the closure motion generated by the closure motor 4603 to an end effector in order to displace a closure tube to close the anvil and compress tissue between the anvil and the staple cartridge. The closure motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue.

[0164] In certain examples, the surgical instrument or surgical device may include, for example, one or more articulation motors 4606a, 4606b. The articulation motors 4606a, 4606b can be operably coupled to their respective articulation motor drive assemblies 4608a, 4608b, which can be configured to transmit the articulation generated by the articulation motors 4606a, 4606b to an end effector. In certain examples, the articulation can cause, for example, the end effector to articulate relative to a shaft.

[0165] In certain examples, the surgical instrument or surgical device may include a control module 4610 that can be used with multiple motors of the surgical instrument 4600. Each of the motors 4602, 4603, 4606a, and 4606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be detected in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0166] In various examples, as shown in Figure 23, the control module 4610 may include a motor driver 4626 which may have one or more H-bridge FETs. The motor driver 4626 can modulate the power transmitted from the power supply 4628 to the motor connected to the control module 4610, for example, based on input from a microcontroller 4620 ("controller"). In certain examples, as described above, the controller 4620 can be used to determine, for example, the current drawn by the motor while the motor is connected to the control module 4610.

[0167] In certain examples, the microcontroller 4620 may include a microprocessor 4622 ("processor") and one or more non-temporary computer-readable media or memory units 4624 ("memory"). In certain examples, the memory 4624 can store various program instructions, which, when executed, cause the processor 4622 to perform some of the functions and / or calculations described herein. In certain examples, one or more memory units 4624 can be linked to the processor 4622, for example. In various examples, the processor 4622 can control a motor driver 4626 to control the position, direction of rotation, and / or speed of a motor linked to a control module 4610.

[0168] In certain examples, one or more mechanisms and / or sensors, such as sensor 4630, may be configured to detect the force (closing force "FTC") applied by the jaws of the end effector of the surgical instrument 4600 to the tissue trapped between the jaws. The FTC can be transmitted to the jaws of the end effector via the closing motor drive assembly 4605. Additionally, or alternatively, sensor 4630 may be configured to detect the force (launching force "FTF") applied to the end effector via the launching motor drive assembly 4604. In various examples, sensor 4630 may be configured to detect closing action (e.g., motor current and FTC), launch action (e.g., motor current and FTF), joint movement (e.g., angular position of the end effector), as well as rotation of the shaft and end effector.

[0169] One or more aspects of process 4000 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4000 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4000. Additionally or alternatively, one or more aspects of process 4000 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4000 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0170] In various embodiments, process 4000 can be implemented by a computer-implemented interactive surgical system 2100 (Figure 19) which includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include a remote server 2113. A control circuit that performs one or more embodiments of process 4000 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can communicate with surgical instruments (e.g., surgical instruments 2112, 4600) and receive instrument data from them. Communication between the surgical instruments and the control circuits of the visualization system may be direct communication, or instrument data may be sent to the visualization system via the surgical hub 2106, for example. In at least one example, a control circuit that performs one or more embodiments of process 4000 may be a component of the surgical hub 2106.

[0171] Referring to Figure 24, in various examples, visualization data 4010 is correlated with instrument data 4011 by constructing a composite dataset 4012 from visualization data 4010 and instrument data 4011. Figure 24 shows the current user's composite dataset 4012 constructed from the current user's visualization data 4010 and the current user's instrument data 4011 in graph 4013. Graph 4013 shows visualization data 4010 representing the first use cycle of surgical instrument 4600, including jaw positioning, clamping, and firing of the surgical instrument 4600. Graph 4013 also shows visualization data 4010 representing the start of the second use cycle of the surgical instrument 4600, in which case the jaws have been repositioned for the second clamping and firing of the surgical instrument 4600. Graph 4013 further shows the current user's instrument data 4011 in the form of FTC data 4014 correlated with clamp visualization data, and FTF data 4015 correlated with firing visualization data.

[0172] As described above, the visualization data 4010 is derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can represent, for example, the distance between the end effector of the surgical instrument 4600 and critical structures in the surgical field during the positioning, clamping, and / or firing of the end effector of the surgical instrument 4600. In at least one example, the visualization system identifies the end effector or its components in the surgical field, identifies critical structures in the surgical field, and tracks the position of the end effector or its components relative to the critical structures or the tissue surrounding the critical structures. In at least one example, the visualization system identifies the jaws of the end effector in the surgical field, identifies critical structures in the surgical field, and further tracks the position of the jaws relative to the critical structures or the tissue surrounding the critical structures in the surgical field.

[0173] In at least one example, the critical structure is a tumor. To remove this tumor, surgeons often want to ensure the entire tumor is removed by cutting the tissue along a safety margin around the tumor. In such an example, visualization data 4010 can represent the distance between the jaws of the end effector and the safety margin of the tumor during the positioning, clamping, and / or firing of the surgical instrument 4600.

[0174] Process 4000 may further include comparing the current user's composite dataset 4012 with another composite dataset 4012' which can be received from an external source and / or derived from a previously collected composite dataset. Graph 4013 shows a comparison between the current user's composite dataset 4012 and another composite dataset 4012' which includes visualization data 4010' and instrument data 4011' with FTC data 4014' and FTF data 4015'. This comparison can be presented in real time to the user of the surgical instrument 4600 in the form of graph 4013 or any other preferred format. A control circuit performing one or more aspects of Process 4000 can compare the two composite datasets on any preferred screen in the operating room, such as a visualization system screen. In at least one example, the comparison can be displayed alongside a real-time video of the surgical field captured on any preferred screen in the operating room. In at least one example, the control circuit is configured to adjust instrument parameters to address deviations detected between the first and second composite datasets.

[0175] Furthermore, control circuits performing one or more aspects of process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) may display the current state of instrument data, such as FTF data and / or FTC data, relative to the best-practice equivalent. In the example shown in Figure 24, the current value of the FTC, represented by circle 4020, is shown in real time relative to gauge 4021, and indicator 4022 represents the best-practice FTC. Similarly, the current value of the FTF, represented by circle 4023, is shown relative to gauge 4024, and indicator 4025 represents the best-practice FTF. Such information can be superimposed in real time onto a video image of the surgical field.

[0176] The example shown in Figure 24 warns the user that the current FTC is higher than the best-practice FTC and that the current FTF is higher than the best-practice FTF. A control circuit performing one or more aspects of process 4000 may warn the current user of the surgical instrument 4600 with an audible, visual, and / or tactile warning mechanism if the current values ​​of the FTF and / or FTC reach and / or exceed predetermined thresholds.

[0177] In a specific example, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4000 can further provide predicted instrument data to the current user of the surgical instrument 4600 based on the current instrument data. For example, as shown in Figure 24, the predicted FTF 4015" is determined based on the current value of the FTF and is further displayed on graph 4013 for the current FTF 4015 and previously collected FTFs. Additionally or alternatively, as shown in Figure 24, a projected FTF circle 4026 can be displayed relative to gauge 4024.

[0178] In various aspects, the previously collected composite dataset, and / or the best practice FTF and / or FTC, are determined from the previous use of surgical instruments 4600 in the same surgical procedures and / or other surgical procedures performed by the user, other users within the hospital, and / or users in other hospitals. Such data can be made available to control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that perform one or more aspects of process 4000, for example, by importing it from the cloud 104.

[0179] In various embodiments, control circuits performing one or more embodiments of process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can visually overlay feedback measurements of tissue thickness, compression, and stiffness onto a screen displaying live video of the surgical instrument 4600 in the surgical field, as the jaws of the end effector begin to deform the tissue trapped between the jaws during the clamping phase. The visual overlay correlates the visualization data representing tissue deformation with changes in clamping force over time. This correlation helps the user confirm the appropriate cartridge selection, determine the firing start time, and determine a suitable firing rate. Furthermore, the correlation can also reveal the adaptive clamping algorithm. Adaptive changes in firing rate can be determined by the measured force, as well as changes in the movement of tissue immediately adjacent to the jaws of the surgical instrument 4600 (e.g., principal strain, tissue slip, etc.), while a gauge or meter communicating the results is overlaid on a screen displaying live video of the end effector in the surgical field.

[0180] In addition to the above, the kinematics of the surgical instrument 4600 can also be used to instruct the instrument's operation to a different user. The kinematics can be verified with an accelerometer, torque sensor, force sensor, motor encoder, or any other suitable sensor, and various force, velocity, and / or acceleration data of the surgical instrument or its components can be acquired and correlated with corresponding visualization data.

[0181] In various embodiments, if visualization data and / or instrument data detect a deviation from the best-practice surgical technique, a control circuit performing one or more embodiments of process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can suggest an alternative surgical technique. In at least one example, an adaptive display of instrument movement, force, tissue impedance, and the results of the proposed alternative technique is shown. If visualization data indicates the detection of blood vessels and clip applicators within the surgical field, a control circuit performing one or more embodiments of process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can further ensure the perpendicularity of the blood vessels to the clip applicators. The control circuit may suggest changing the position, orientation, and / or roll angle to achieve the desired perpendicularity.

[0182] In various embodiments, control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can perform process 4000 by comparing real-time visualization data with a preoperative planning simulation. Users can use preoperative patient scans to simulate surgical approaches. The preoperative planning simulation can enable users to follow a specific preoperative plan based on training runs. The control circuits can be configured to correlate reference landmarks from the preoperative scan / simulation with the current visualization data. In at least one example, the control circuits may employ object boundary tracking to establish the correlation.

[0183] When a surgical instrument interacts with tissue and deforms its surface geometry, the change in surface geometry can be calculated as a function of the instrument's position. For a given change in the instrument's position during tissue contact, the corresponding change in the tissue's geometry may depend on the subsurface structure within the tissue region in contact with the instrument. For example, in thoracic surgery, the change in tissue geometry in a region containing substructures of the airway differs from that in a region containing substructures of solid tissue. Generally, the stiffer the substructure, the smaller the change in surface tissue geometry in response to a given change in the instrument's position. In various embodiments, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to calculate the runnable average of the change in instrument position for a given patient's surface geometry change and obtain patient-specific differences. Additionally or alternatively, the calculated runnable average can be compared to a second set of previously collected data. In certain cases, if changes in surface geometry are not measured for each change in instrument position, a surface reference may be selected. In at least one example, the control circuit may be configured to determine the position of a substructure based on detected changes in surface geometry in response to a given contact between a tissue area and a surgical instrument.

[0184] Furthermore, the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain contact between the set instrument and the tissue throughout tissue treatment, based on a correlation between the contact between the set instrument and the tissue and one or more changes in the tissue and surface geometry associated with the contact between the set instrument and the tissue. For example, the end effector of surgical instrument 4600 can clamp the tissue between its jaws with a desired compression to set contact between the instrument and the tissue. The corresponding changes in the tissue and surface geometry can be detected by a visualization system. Furthermore, the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visualization data showing the changes in the tissue and surface geometry associated with the desired compression. Furthermore, the control circuits can automatically adjust the motor settings of the closing motor 4603 (Figure 22) to maintain the changes in the tissue and surface geometry associated with the desired compression. In this configuration, continuous interaction between the surgical instrument 4600 and the visualization system is required to maintain changes in tissue and surface geometry associated with the desired compression by continuously adjusting the compression of the jaws on the tissue based on visualization data.

[0185] In yet another example, if the surgical instrument 4600 is a robotic tool attached to the robotic arm of a robotic surgical system (e.g., robotic system 110), the robotic surgical system can be configured to automatically adjust one or more components of the robotic surgical system to maintain the set surface contact with the tissue, based on visualization data derived from detected changes in the surface geometry of the tissue detected in response to the set surface contact with the tissue.

[0186] In various applications, visualization data is used in conjunction with measured instrument data to maintain contact between tissues, either through positional or load control, allowing the user to manipulate the tissue and apply a predetermined load while the instrument is moved across it. The user can specify which contact or pressure they wish to maintain, and visual tracking of the instrument, along with its internal load, may allow for repositioning without changing fixed parameters.

[0187] Referring to Figures 25A and 25B, the screen 4601 of the visualization system (e.g., visualization systems 100, 160, 500, 2108) displays real-time video footage of the surgical field during surgery. The end effector 4642 of the surgical instrument 4600 includes jaws for clamping tissue near a tumor identified in the surgical field, for example, via superimposed MRI images. The jaws of the end effector 4642 comprise an anvil 4643 and a channel for housing a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other to capture tissue between the anvil 4643 and the staple cartridge. The captured tissue is then stapled via staples 4644 that are deployable from the staple cartridge during the firing sequence of the surgical instrument 4600. In addition, the captured tissue is cut via a distally advanced cutting member 4645 during the firing sequence, but shortly after the staple deployment.

[0188] As is evident in Figure 25A, during the firing sequence, the position and / or movement of the captured tissue and certain internal components of the end effector 4642, such as the staple 4644 and the cutting member 4645, may not be visible in the standard view 4640 of the live image on the screen 4601. Certain end effectors include windows 4641, 4653 that partially reveal the cutting member 4645 at the beginning and end of the firing sequence, but conceal it during the firing sequence. Therefore, the user of the surgical instrument 4600 cannot track the progress of the firing sequence on the screen 4601.

[0189] Figure 26 is a logic flowchart of process 4030, which shows a control program or logic configuration for synchronizing the movement of a virtual representation of an end-effector component with the actual movement of the end-effector component, relating to at least one aspect of the present disclosure. Process 4030 is generally performed during a surgical operation and includes, 4031, detecting the movement of an internal component of the end-effector during the firing sequence; 4032, presenting a virtual representation of this internal component on the end-effector, for example, by overlaying it; and 4033, synchronizing the movement of the virtual representation on screen 4601 with the detected movement of the internal component.

[0190] One or more aspects of process 4030 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4030 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4030 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4030 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0191] In various examples, control circuits performing one or more aspects of process 4030 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can receive instrument data indicating the movement of internal components of the end effector 4642 during the firing sequence. The movement of the internal components can be tracked, for example, using a conventional rotary encoder of the firing motor 4602. In other examples, the movement of the internal components can be tracked by a tracking system employing an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In some examples, the movement of the internal components can be tracked using one or more position sensors, which may comprise any number of magnetic sensing elements, such as magnetic sensors classified depending on whether they measure the total magnetic field or the vector component of the magnetic field.

[0192] In various embodiments, process 4030 includes an overlay trigger. In at least one example, the overlay trigger can detect that tissue has been captured by the end effector 4642. When tissue captured by the end effector 4642 is detected, process 4030 superimposes a virtual representation of the cutting member 4645 at its starting position onto the end effector 4642. Process 4030 further includes projecting a staple line outlining where the staples will unfold within the captured tissue. Furthermore, in response to the user initiating a firing sequence, process 4030 moves the virtual representation of the cutting member 4645 distally, mimicking the actual movement of the cutting member 4645 within the end effector 4642. Once the staples are unfolded, process 4030 converts the unfired staples to fired staples, allowing the user to visually track the staple unfolding and the forward movement of the cutting member 4645 in real time.

[0193] In various examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4030 can detect that tissue has been captured by the end effector 4642 from instrument data indicating the force applied by the closing motor 4603 (Figure 22) to the jaws of the end effector 4642 through the closing motor drive assembly 4605. The control circuit can further determine the position of the end effector in the surgical field from visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the position of the end effector can be determined relative to a reference point in the tissue, such as a critical structure.

[0194] In any case, the control circuit superimposes a virtual representation of the internal components onto the end effector 4642 on screen 4601 at a position corresponding to the location of the internal components within the end effector. Furthermore, the control circuit moves the projected virtual representation of the internal components in synchronization with the internal components during the firing sequence. In at least one example, synchronization is improved by the integration of markings on the end effector 4642, which the control circuit can use as a reference point to determine where the virtual representations overlap.

[0195] Figure 25B shows an extended view 4651 of the live image of the surgical field on screen 4601. In the extended view 4651 of the example in Figure 25B, virtual representations of staples 4644 and cutting members 4645 are superimposed on the end effector 4642 during the firing sequence. The overlay tracks the progress of the firing sequence by distinguishing fired staples 4644a from unfired staples 4644b and distinguishing completed cutting lines 4646a from projected cutting lines 4646b. Furthermore, the overlay also shows the starting point of the staple line 4647 and the projected end 4649 of the staple line at cutting lines that do not reach the edge of the tissue. Then, based on the overlay of the tumor MRI image, the safety margin distance "d" between the tumor and the projected cutting line 4646b is measured and presented together with the overlay. The superimposed safety margin distance "d" assures the user that all tumor is removed.

[0196] As shown in Figure 25B, the control circuit is configured to cause the visualization system to continuously rearrange virtual representations of the internal components in order to correlate them with the actual movement of the internal components. In the example in Figure 25B, this overlay shows that the completed cutting line 4646 is lagging slightly behind the fired staple line 4644a by a distance "d1", which assures the user that the firing sequence is proceeding correctly.

[0197] Referring here to Figure 27, a visualization system (e.g., visualization systems 100, 160, 500, 2108) employs fixture illumination 4058 and camera 4059 to detect and / or define the trocar position. Based on the determined trocar position, the user can be directed to the trocar port best suited to completing the intended function, based on time efficiency, location of critical structures, and / or avoidance or risk.

[0198] Figure 27 shows three trocar positions (trocar 1, trocar 2, and trocar 3) extending through the body wall 4050 at different positions and orientations relative to the body wall and to a critical structure 4051 within a cavity 4052 inside the body wall 4050. The trocars are represented by arrows 4054, 4055, and 4056. The trocar positions can be detected by using cascading light or images from the surrounding environment via the illuminator 4058. Furthermore, the light source of the illuminator 4058 may be a rotating light source. In at least one example, the light sources of the illuminator 4058 and the camera 4059 are used to detect the distance of the trocars to a target position, such as the critical structure 4051. In various examples, once a more preferred fixture position is determined based on visualization data derived from the camera 4059 recording the light projected by the light source of the illuminator 4058, the visualization system can indicate changes in the fixture's position. Screen 4060 can display the distance between the trocar and the target tissue, whether instrument access through the trocar is permissible, the risks associated with the use of the trocar, and / or the expected surgical time using the trocar, which can assist the user in selecting the optimal trocar for introducing surgical instruments into the cavity 4052.

[0199] In various embodiments, the surgical hub (e.g., surgical hubs 2106, 2122) can recommend the optimal trocar for inserting a surgical instrument into the cavity 4052 based on user characteristics that can be received, for example, from a user database. User characteristics include the user's dominant hand, the patient-side user's preference, and / or the user's physical characteristics (e.g., height, arm length, range of motion). The surgical hub can use these characteristics, data on the position and orientation of available trocars, and / or the position data of critical structures to select the optimal trocar for inserting the surgical instrument in order to reduce user fatigue and increase efficiency. In various embodiments, the surgical hub can further reverse control of the surgical instrument if the user reverses the orientation of the end effector.

[0200] The surgical hub can reconfigure the output of surgical instruments based on visualization data. For example, if visualization data indicates that a surgical instrument has been retracted or is being used to perform a different task, the surgical hub can suppress the activation of the therapeutic energy output of the surgical instrument.

[0201] In various embodiments, visualization systems can be configured to track the blood surface or estimate blood volume-based reflected IR or red wavelengths to depict blood from non-blood surface and surface geometric shape measurements. This can be reported as an absolute static index or as a rate of change to provide quantitative data on the amount and degree of change of bleeding.

[0202] Referring to Figure 28, for example, various elements of a visualization system (e.g., visualization systems 100, 160, 500, 2108), such as a structured light projector 706 and a camera 720, can be used to generate visualization data of anatomical organs and generate a virtual 3D structure 4130 of the anatomical organs.

[0203] As described herein, structured light, for example, in the form of stripes or lines, can be projected from a light source and / or projector 706 onto the surface 705 of a targeted anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which may be similar in various respects to the imaging device 120 (Figure 1), can be configured to detect the projected light pattern on the surface 705. As the projected pattern deforms upon impact with the surface 705, the visual system can calculate the depth and surface information of the targeted anatomical structure.

[0204] Figure 29 is a logic flow diagram of a process 4100 showing a control program or logic configuration relating to at least one aspect of the present disclosure. In various examples, the process 4100 identifies a surgical procedure in 4101 and identifies the anatomical organ targeted by the surgical procedure in 4102. Furthermore, the process 4100 generates a virtual 3D structure 4130 of at least a portion of the anatomical organ in 4104, identifies the anatomical structure of at least a portion of the anatomical organ related to the surgical procedure in 4105, links the anatomical structure to the virtual 3D structure 4130 in 4106, and overlays a surgical layout plan determined based on the anatomical structure onto the virtual 3D structure 4130 in 4107.

[0205] One or more embodiments of process 4100 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4100 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4100. Additionally or alternatively, one or more embodiments of process 4100 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, one or more embodiments of process 4100 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0206] In various aspects, process 4100 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. A control circuit that executes one or more aspects of process 4100 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0207] A control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4100 can identify a surgical procedure at 4101 and / or identify an anatomical organ targeted by the surgical procedure at 4102 by obtaining such information from a database storing the information or directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). In at least one example, the database includes an in-hospital EMR.

[0208] In one embodiment, the surgical system 2200 includes a surgical hub 2236 connected to multiple surgical site devices, such as visualization systems (e.g., visualization systems 100, 160, 500, 2108) located in the operating room. In at least one example, the surgical hub 2236 includes a communication interface for communicating with the visualization systems, the cloud 2204, and / or a remote server 2213. A control circuit of the surgical hub 2236 performing one or more embodiments of process 4100 can, 4101, identify a surgical procedure and / or, 4102, identify an anatomical organ targeted by the surgical procedure, by retrieving such information from a database stored in the cloud 2204 or from the remote server 2213.

[0209] A control circuit performing one or more aspects of process 4100 may cause a visualization system (e.g., visualization systems 100, 160, 500, 2108) to perform an initial scan of at least a portion of an anatomical organ to generate a three-dimensional ("3D") structure 4130 of at least a portion of the anatomical organ targeted for surgical intervention in 4104. In the example shown in Figure 28, the anatomical organ is the stomach 4110. The control circuit may cause one or more elements of the visualization system, such as a structured light projector 706 and a camera 720, to generate visualization data by using structured light 4111 to perform a scan of at least a portion of the anatomical organ when the camera(s) are introduced into the body. The 3D structure of at least a portion of the anatomical organ can be generated by utilizing current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), visualization data from previous similar surgeries performed on the same or other patients, and / or user input.

[0210] Further, a control circuit that executes one or more aspects of process 4100, at 4105, identifies the anatomical structure of at least a portion of an anatomical organ related to a surgical procedure. In at least one example, the user can select the anatomical structure using any suitable input device. Additionally or alternatively, the visualization system includes one or more imaging devices 120 with a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect a reflected spectral waveform and generate an image based on molecular responses for various wavelengths. The control circuit can utilize the optical absorption characteristics or refractive characteristics of tissue to distinguish different tissue types of the anatomical organ, thereby identifying the relevant anatomical structure. Additionally, the control circuit can utilize current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input to identify the relevant anatomical organ.

[0211] The identified anatomical structure can be the anatomical structure of the surgical field, and / or the anatomical structure can be selected by the user. In various examples, the position tracking of the relevant anatomical structure can be extended beyond the current view of a camera oriented towards the surgical field. In one example, this is achieved by using a common visual connection landmark or by using secondary connection motion tracking. The secondary tracking can be achieved, for example, through a secondary imaging source, calculation of the movement of the scope, and / or a pre-established beacon that is a measurement by a secondary visualization system.

[0212] As detailed above in relation to Figure 14, the visualization system can use the structured light projector 706 to cast an array of patterns or lines that the camera 720 can use to determine the distance to the target location. The visualization system can then emit a pattern or lines of known size at a set distance equal to the determined distance. In addition, the spectral camera can determine the size of the pattern, which may vary depending on the light absorption or refractive properties of the tissue at the target location. The difference between the known size and the determined size indicates the tissue density at the target location, which indicates the tissue type at the target location. A control circuit performing one or more aspects of process 4100 can identify relevant anatomical structures, at least partially based on the determined tissue density at the target location.

[0213] In at least one example, the detected abnormality in tissue density may be related to a disease state. Furthermore, the control circuit selects, updates, or modifies one or more settings of the surgical instrument used to treat the tissue based on the tissue density detected via visualization data. For example, the control circuit may change various clamp parameters and / or firing parameters of a surgical stapler used to staple and cut tissue. In at least one example, the control circuit may allow the firing sequence to be slowed down and / or the clamp time to be extended based on the tissue density detected by visualization data. In various examples, the control circuit may warn the user of the surgical instrument of abnormal tissue density by displaying on the screen commands, for example, commands to reduce the byte size, commands to increase or decrease the energy delivery output of the electrosurgical instrument, and commands to adjust the amount of jaw closure. In another example, if visualization data indicates that the tissue is adipose tissue, the command may be to increase the power to shorten the energy application time.

[0214] Furthermore, identifying the type of surgical procedure facilitates target organ identification by the control circuit. For example, if the procedure is a left upper lobectomy, the lung is likely the target organ. Therefore, the control circuit considers only the visualized and non-visible data related to the lung and / or instruments commonly used in such procedures. Knowledge of the type of procedure also makes other image fusion algorithms more effective, for example, informing the location of tumors or the placement of staple lines.

[0215] In various embodiments, knowledge of the operating table position and / or pneumoperitoneum pressure can be used by a control circuit performing one or more embodiments of process 4100 to establish the baseline position of the target anatomical organ and / or related anatomical structure identified from the visualization data. Movement of the operating table (e.g., moving the patient from a flat position to the inverted Trendelenburg position) can cause deformation of anatomical structures, which can be tracked and compared to the baseline to continuously inform the position and state of the target organ and / or related anatomical structure. Similarly, changes in intracavitary pneumoperitoneum pressure may interfere with the baseline visualization data of the target organ and / or related anatomical structure within the body cavity.

[0216] The control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more embodiments of the process of deriving baseline visualization data of the patient's target organ and / or related anatomical structures on the operating table during surgical procedures, determining changes in the position of the operating table, and re-deriving baseline visualization data of the patient's target organ and / or related anatomical structures at the new position.

[0217] Similarly, control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more embodiments of the process of deriving baseline visualization data of the patient's target organ and / or related anatomical structures on the operating table during surgery, determining changes in the patient's intracavitary pressure, and re-deriving baseline visualization data of the patient's target organ and / or related anatomical structures at the new insufflation pressure.

[0218] In various examples, as shown in Figure 28, a control circuit performing one or more embodiments of process 4100 can link identified anatomical structures to a virtual 3D structure by superimposing landmarks or markers onto the virtual 3D structure of the organ to indicate the location of the anatomical structure. The control circuit can also superimpose user-defined structure and tissue planes onto the virtual 3D structure. In various embodiments, a hierarchy of tissue types can be established to organize the identified anatomical structures on the virtual 3D structure. Table 1, provided below, shows an exemplary hierarchy for the lungs and stomach.

[0219] [Table 1]

[0220] In various embodiments, relevant anatomical structures identified on the virtual 3D structure can be renamed and / or rearranged by the user to correct errors or, at their preference. In at least one example, the correction may be voice-activated. In at least one example, the correction is recorded for future machine learning.

[0221] In addition to the above, control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that perform one or more aspects of process 4100 can, in 4107, overlay a surgical layout plan (e.g., layout plan 4120) onto a virtual 3D structure of the target organ (e.g., stomach 4110). In at least one example, the virtual 3D structure is displayed on a separate screen of the visualization system from a screen displaying live video / view of the surgical field. In another example, one screen can alternately display live video of the surgical field and live video of the 3D structure. In such an example, the user can switch between the two views using any preferred input device.

[0222] In the example shown in Figure 28, the control circuit determines that the surgical procedure is a sleeve gastrectomy and the target organ is the stomach. During the initial scan of the abdominal cavity, the control circuit identifies the stomach, liver, spleen, greater curvature of the stomach, and pylorus using visualization data such as structured optical data and / or spectral data. This is informed by knowledge of the procedure and the target structure.

[0223] For example, visualization data such as structured light data and / or spectral data can be used by a control circuit to identify the stomach 4110 by comparing the current structured light data with stored structured light data previously associated with such an organ. In at least one embodiment, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can use structured light data representing the characteristic anatomical contours of the organ and / or spectral data representing characteristic subsurface tissue features to identify anatomical structures relevant to a surgical layout plan (e.g., layout plan 4120).

[0224] In at least one example, the visualization data can be used to identify the pyloric vein 4112 indicating the location 4113 of the pylorus 4131, the gastrocolic vessel 4114 indicating the location 4115 of the greater curvature of the stomach 4110, the curve 4116 of the right gastric vein indicating the location 4117 of the notch angle 4132, and / or the location 4119 of the His angle 4121. The control circuit can assign landmarks to one or more of the identified locations. In at least one example, as shown in Figure 28, the control circuit causes the visualization system to overlay the landmarks onto locations 4113, 4117, and 4119 on the virtual 3D structure of the stomach 4110 generated using the visualization data as described above. In various embodiments, the landmarks can be overlaid synchronously on the virtual 3D structure and the surgical field view, allowing the user to switch views without losing sight of the landmarks. The user can zoom in on the screen view displaying the virtual 3D structure to show the overall layout plan, or zoom in to show a portion similar to the surgical field view. The control circuit can continuously track and update landmarks.

[0225] In sleeve gastrectomy, the surgeon primarily staples the gastric tissue approximately 4 cm from the pylorus. Prior to stapling, an energy device is introduced into the patient's abdominal cavity at the start of the sleeve gastrectomy, and the gastroepiploic artery and peritoneum are incised from the greater curvature at a position approximately 4 cm from the pylorus or approximately 4 cm from the pylorus. As described above, the control circuit that identifies position 4113 can automatically cause an overlay of the energy device's end effector at a position approximately 4 cm from position 4113 or approximately 4 cm from the pylorus. By superimposing the end effector of the energy device at a position approximately 4 cm from the pylorus or approximately 4 cm from the pylorus, or any suitable landmark, the starting position of the sleeve gastrectomy can be determined.

[0226] As the surgeon makes an incision along the greater curvature of the stomach, the control circuit causes the superimposed end effector of the landmark and / or energy device at position 4113 to disappear. When the surgeon approaches the spleen, a distance indicator is automatically superimposed on the virtual 3D structure view and / or surgical field view. The control circuit can cause the distance indicator to pinpoint a distance of 2 cm from the spleen. The control circuit can cause the distance indicator to flash and / or change color when the incision path reaches, or is about to reach, for example, 2 cm from the spleen. The distance indicator overlay remains until the user reaches position 4119 of angle His 4121.

[0227] Referring to Figure 30, once a surgical stapler is introduced into the abdominal cavity, the control circuit can use visualization data to identify the pylorus 4131, the angular notch 4132, the greater curvature 4133 of the stomach 4110, the lesser curvature 4134 of the stomach 4110, and / or other anatomical structures associated with sleeve gastrectomy. A bougie overlay may also be shown. The introduction of a surgical instrument into a body cavity (e.g., the introduction of a surgical stapler into the abdominal cavity) can be detected by the control circuit from visualization data indicating a visual cue on the end effector, such as a unique color, marking, and / or shape. The control circuit can identify such a visual cue and the surgical instrument in a database that stores the corresponding visual cues. Alternatively, the control circuit can prompt the user to identify the surgical instrument inserted into the body cavity. Alternatively, a surgical trocar assisting access to a body cavity may include one or more sensors for detecting surgical instruments inserted through it. In at least one example, the sensors comprise an RFID reader configured to identify the surgical instrument from the RFID chip on the surgical instrument.

[0228] In addition to landmarks that identify relevant anatomical structures, the visualization system can also overlay a surgical procedure layout plan 4135, which may be in the form of a recommended treatment pathway, onto the 3D critical structures and / or the operative field view. In the example of FIG. 30, the surgery is a sleeve gastrectomy, and the treatment layout plan 4135 is in the form of three resection paths 4136, 4137, 4138 and the corresponding outcome volume of the resulting sleeve.

[0229] As shown in FIG. 30, it is the distance (a, a1, 2) from the pylorus 4131 to the starting point for creating the sleeve. The sleeve sizes resulting from each starting point are different (e.g., for starting points 4146, 4147, 4148 at distances a, a1, a2 from the pylorus 4131, they are 400 cc, 425 cc, 450 cc respectively). In one example, the control circuit prompts the user for a size selection input and, in response, presents a treatment layout plan that can be in the form of a resection path resulting in the selected sleeve size. In another example, as shown in FIG. 30, the control circuit presents a plurality of resection paths 4136, 4137, 4138 and the corresponding sleeve sizes. Then, the user can select one of the proposed resection paths 4136, 4137, 4138, and in response, the control circuit removes the unselected resection paths.

[0230] In yet another example, the control circuit enables the user to adjust the proposed resection path on a screen showing the resection path overlaid on the virtual 3D structure and / or the operative field. The control circuit can calculate the sleeve size based on the adjustment. Alternatively, in another example, the user can select a starting point for forming the sleeve at a desired distance from the pylorus 4131. In response, the control circuit calculates the sleeve size based on the selected starting point.

[0231] For example, the visualization system can achieve the presentation of the resection route by overlaying it onto a virtual 3D structure view and / or surgical field view. Conversely, the visualization system can achieve the removal of the proposed resection route by removing such an overlay from the virtual 3D structure view and / or surgical field view.

[0232] Referring further to Figure 30, in a particular embodiment, when the end effector of the surgical stapler clamps the gastric tissue between a starting point selected from the proposed starting points 4146, 4147, 4147 and an ending position 4140 at a predetermined distance from the notch angle 4132, the control circuit presents information regarding the clamping and / or firing of the surgical stapler. In at least one example, a composite dataset 4012 can be displayed from visualization data 4010 and instrument data 4011, as shown in Figure 24. Additionally, or alternatively, the values ​​of FTC and / or FTF can be displayed. For example, the current value of FTC, represented by circle 4020, is shown in real time against gauge 4021, and indicator 4022 can represent the best-performing FTC. Similarly, the current value of FTF, represented by circle 4023, is shown against gauge 4024, and indicator 4025 can represent the best-performing FTF.

[0233] After the surgical stapler is fired, a recommendation for selecting a new cartridge can be displayed on the screen of the surgical stapler or on one of the screens of the visualization system, as detailed below. Once the surgical stapler is removed from the abdominal cavity, reloaded with the selected staple cartridge, and reintroduced into the abdominal cavity, a distance indicator (d) that identifies a certain distance from multiple points along the lesser curvature 4134 of the stomach 4110 to the selected resection route is superimposed on the virtual 3D structural view and / or surgical field view. To ensure proper orientation of the end effector of the surgical stapler, the distance from the target at the distal end of the end effector of the surgical stapler, as well as the distance from the proximal end to the previously fired staple line, are superimposed on the virtual 3D structural view and / or surgical field view. This process is repeated until the resection is complete.

[0234] One or more of the distances proposed and / or calculated by the control circuit can be determined based on stored data. In at least one example, the stored data includes preoperative data, user preference data, and / or data on surgical procedures previously performed by the user or other users.

[0235] Referring to Figure 31, process 4150 represents a control program or logical configuration for proposing a resection route for removing a portion of an anatomical organ, relating to at least one aspect of the present disclosure. Process 4150, as described in more detail elsewhere in this specification in relation to process 4100, identifies the anatomical organ to be targeted for surgery in 4151, identifies the anatomical structure of the anatomical organ related to the surgery in 4152, and proposes a resection route for removing a portion of the anatomical organ using surgical instruments in 4153. The surgical resection route is determined based on the anatomical structure. In at least one example, the surgical resection route includes different starting points.

[0236] One or more embodiments of process 4150 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4150 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4150. Additionally or alternatively, one or more embodiments of process 4150 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4150 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0237] Referring to Figures 32A to 32D, control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4100 or process 4150 may utilize dynamic visualization data to update or modify the surgical layout plan in real time during implementation. In at least one example, the control circuit modifies a pre-configured resection route (Figure 32B) for removing a portion of an organ or an abnormality (e.g., a tumor or site) by a surgical instrument to an alternative resection route (Figure 32D) based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization systems 100, 160, 500, 2108) that tracks the progress of the tissue to be resected and surrounding tissue. Modification of the resection route can be triggered by displacement of important structures (e.g., blood vessels) into the resection route. For example, the tissue resection process can sometimes lead to inflammation of the tissue, altering its shape and / or volume, which can cause displacement of important structures (e.g., blood vessels). Dynamic visualization data allows the control circuit to detect changes in the position and / or volume of important structures and / or related anatomical structures near the pre-defined resection path. If the changes in position and / or volume cause important structures to shift into or within a safety margin of the resection path, the control circuit modifies the pre-defined resection path by selecting or at least recommending an alternative resection path for the surgical instrument.

[0238] Figure 32A shows a live view 4201 of the surgical field on the screen 4230 of the visualization system. Surgical instruments 4200 are introduced into the surgical field to remove the target region 4203. As shown in the magnified view of region 4203 in Figure 32B, the initial planning layout 4209 for removing the region is superimposed on the live view 4201. Region 4203 is surrounded by critical structures 4205, 4206, 4207, and 4208. As shown in Figure 32B, the initial planning layout 4209 extends the resection path from region 4203, around region 4203, within a predetermined safety margin. The resection path avoids crossing or passing over critical structures by extending either outside (e.g., critical structure 4208) or inside (e.g., critical structure 4206) of the critical structures. As described above, the initial planning layout 4209 is determined by the control circuit based on visualization data from the visualization system.

[0239] Figure 32A shows a live view 4201' of the surgical field on the visualization system screen 4230 at a later time (00:43). The end effector 4202 of the surgical instrument 4200 excises tissue along a predetermined resection route defined by the layout plan 4209. Due to changes in tissue volume, including areas 4203 caused by tissue inflammation, critical structures 4206 and 4208, for example, shift to the predetermined resection route. In response, as shown in Figure 32D, the control circuit proposes an alternative resection route 4210 that navigates around critical structures 4206 and 4208, protecting them from damage. In various examples, alternative resection routes can be proposed to minimize bleeding, shorten surgical time, reduce the pressure of dealing with unexpected situations, and balance the impact on the volume of remaining organs by providing the user with guidance to optimize the amount of healthy tissue to be left and ensure that critical structures are not hit.

[0240] In various embodiments, a control circuit performing one or more embodiments of one or more processes described herein may receive and / or derive visualization data from multiple imaging devices of a visualization system. The visualization data facilitates tracking of critical structures outside the live view of the surgical field. Common landmarks may enable the control circuit to integrate visualization data from multiple imaging devices of the visualization system. In at least one example, secondary tracking of critical structures outside the live view of the surgical field may be achieved through a secondary imaging source, calculation of scope movement, or a pre-established beacon / landmark measured by a second system.

[0241] Referring in general to Figures 33 to 35, the logical flow diagram of process 4300 illustrates a control program or logical configuration for presenting or superimposing parameters of surgical instruments to or near a proposed surgical resection route, relating to at least one aspect of the present disclosure. Process 4300 is generally performed during surgery and includes, 4301, identifying the anatomical organ targeted for surgery; 4302, identifying anatomical structures related to the surgery from visualization data from at least one imaging device; and 4303, proposing a surgical resection route for removing a portion of the anatomical organ with surgical instruments. In at least one example, the surgical resection route is determined based on the anatomical structure. Process 4300 further includes, 4304, presenting parameters of surgical instruments according to the surgical resection route. Additionally or alternatively, process 4300 further includes, 4305, adjusting parameters of surgical instruments according to the surgical resection route.

[0242] One or more aspects of process 4300 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4300 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4300 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4300 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0243] In various examples, a control circuit performing one or more aspects of process 4300 (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) may, as described elsewhere in this specification in relation to processes 4150 (Figure 31) and 4100 (Figure 29), in 4301 identify an anatomical organ targeted for surgical intervention, in 4302 identify anatomical structures related to the surgical intervention from visualization data from at least one imaging device of a visualization system (e.g., visualization systems 100, 160, 500, 2108), and / or in 4303 propose a surgical resection route for removing a portion of the anatomical organ by a surgical instrument (e.g., surgical instrument 4600). Furthermore, a control circuit performing one or more aspects of process 4300 may propose or recommend one or more parameters of a surgical instrument according to the surgical resection route proposed in 4303. In at least one example, the control circuit presents recommended parameters for the surgical instrument by superimposing such parameters on or near the proposed surgical path, as shown in Figures 34 and 35, in 4304.

[0244] Figure 34 shows a virtual 3D structure 4130 of the stomach of a patient undergoing sleeve gastrectomy performed using surgical instrument 4600, according to at least one aspect of the present disclosure. As will be described in more detail in relation to Figure 28, the virtual 3D structure 4130 can be generated by creating visible data using various elements of a visualization system (e.g., visualization systems 100, 160, 500, 2108), such as a structured light projector 706 and a camera 720. Relevant anatomical structures (e.g., pylorus 4131, angular notch 4132, angle of His 4121) are identified from visualization data from one or more imaging devices of the visualization system. In at least one example, landmarks are assigned to the locations 4113, 4117, 4119 of such anatomical structures by superimposing the landmarks onto the virtual 3D structure 4130.

[0245] In addition, 4303 proposes a surgical resection route 4312 based on the identified anatomical structures. In at least one example, as shown in Figure 34, the control circuit superimposes the surgical resection route 4312 onto a virtual 3D structure 4130. As will be described in more detail elsewhere in this specification, the proposed surgical route can be automatically adjusted based on the desired volume output. In addition, it is automatically adjustable to the projected margins based on important structures and / or tissue abnormalities automatically identified by the control circuit from the visualization data.

[0246] In various embodiments, a control circuit performing at least one embodiment of process 4300 presents parameters 4314 for a surgical instrument selected according to the surgical resection route 4312 proposed in 4303. In the example shown in Figure 34, the parameters 4314 indicate a staple cartridge automatically selected for use with the surgical instrument 4600 when performing a sleeve gastrectomy based on the surgical resection route proposed in 4303. In at least one embodiment, the parameters 4314 include at least one of the staple cartridge size, staple cartridge color, staple cartridge type, and staple cartridge length. In at least one embodiment, the control circuit presents recommended parameters 4314 for the surgical instrument 4600 in 4304 by superimposing such parameters on or near the surgical route 4312 proposed in 4303, as shown in Figures 34 and 35.

[0247] In various embodiments, a control circuit performing at least one embodiment of process 4300 presents tissue parameters 4315 along one or more portions of the surgical resection route 4312. In the example shown in Figure 34, the tissue parameter 4315 is the tissue thickness, presented by displaying a cross-section along line AA, which represents the tissue thickness along at least one portion of the surgical resection route 4312. In various embodiments, the staple cartridges used by the surgical instrument 4600 may be selected according to the tissue parameters 4315. For example, as shown in Figure 34, a black cartridge with larger staple sizes is selected for use with thicker muscle tissue of the vestibule, and a green cartridge with smaller staple sizes is selected for use with myocardial tissue of the body and the gastric fundus of the stomach.

[0248] The tissue parameter 4315 includes at least one of the tissue thickness, tissue type, and sleeve volume outcome resulting from the proposed surgical resection route 4312. The tissue parameter 4315 can be derived from previously acquired CT, ultrasound, and / or MRI images of the patient's organ, and / or from previously known mean tissue thickness. In at least one example, the surgical instrument 4600 is an intelligent instrument (similar to the intelligent instrument 2112), and the tissue thickness and / or selected staple cartridge information is transmitted to the surgical instrument 4600 to optimize the closure setting, firing setting, and / or any other preferred surgical instrument setting. In one example, as described in relation to Figures 17-19, the tissue thickness and / or selected staple cartridge information can be transmitted to the surgical instrument 4600 from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and a surgical hub (e.g., surgical hubs 2106, 2122) that communicates with the surgical instrument 4600.

[0249] In various examples, a control circuit performing at least one aspect of process 4300 proposes arrangements 4317 of two or more staple cartridge sizes (e.g., 45 mm and 60 mm) according to a determined tissue thickness along at least a portion of the surgical resection path 4312. Furthermore, as shown in Figure 35, the control circuit may also present arrangements 4317 along the surgical resection path 4312 proposed in 4303. Alternatively, the control circuit can present a suitable arrangement 4317 along a surgical resection path selected by the user. As described above, the control circuit can determine the tissue thickness along the user-selected resection path and propose staple cartridge arrangements according to this tissue thickness.

[0250] In various embodiments, a control circuit performing one or more embodiments of process 4300 may propose or optimize a selected surgical resection route to minimize the number of staple cartridges in the staple cartridge arrangement 4317 without the resulting sleeve size being compromised below a predetermined threshold. Reducing the number of spent cartridges reduces procedure time and cost, and reduces patient trauma.

[0251] Continuing to refer to Figure 35, arrangement 4317 includes a first staple cartridge 4352 and a final staple cartridge 4353 that define the start and end of the surgical resection route 4312. If only a small portion of the final staple cartridge 4353 of the proposed staple cartridge arrangement 4317 is needed, the control circuit can adjust the surgical resection route 4312 to eliminate the need for the final staple cartridge 4353 without the resulting sleeve size being reduced below a predetermined threshold.

[0252] In various examples, a control circuit performing at least one aspect of process 4300 presents a virtual firing of the proposed staple cartridge arrangement 4317, which virtually separates the virtual 3D structure 4130 into a retaining section 4318 and a removal section 4319, as shown in Figure 35. The retaining section 4318 is a virtual representation of the sleeve resulting from the implementation of the proposed surgical resection route 4312 by firing the staple cartridge arrangement 4317. The control circuit can further determine the estimated volumes of the retaining section 4318 and / or the removal section 4319. The volume of the retaining section 4318 represents the volume of the resulting sleeve. In at least one example, the volumes of the retaining section 4318 and / or the removal section 4319 are derived from visualization data. In another example, the volumes of the retaining section 4318 and / or the removal section 4319 are determined from a database storing the retaining volume, the removal section volume, and the corresponding surgical resection route. The database can be constructed from previously performed surgical procedures on organs that were resected to the same or at least similar dimensions and via the same or at least similar resection routes.

[0253] In various examples, a predetermined combination of average tissue thickness data based on organ context recognition, as described in more detail above, can be used in combination with volumetric analysis from a visualization source, as well as secondary ultrasound imaging, if available for the patient, from CT, MRI, and / or other sources, to select the first staple cartridge for configuration 4317. Subsequent staple cartridge firings within configuration 4317 can have their firing parameters optimized using instrument data from previous firings, in addition to visualization data. Instrument data that may be used to supplement volumetric measurements may include, for example, FTF, FTC, pull-in current by the motor driving firing and / or closure, closure gap of the end effector, firing speed, tissue impedance measurements across the jaws, and / or waiting or pause time during use of the surgical instrument.

[0254] In various examples, visualization data, such as structured optical data, can be used to track changes in the surface geometry of tissue being treated by a surgical instrument (e.g., surgical instrument 4600). Furthermore, visualization data, such as spectral data, can be used to track important structures beneath the tissue surface. Structured data and / or spectral data can be used to maintain contact between the set instrument and the tissue throughout tissue treatment.

[0255] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between its jaws. For example, once contact between the desired tissue and the instrument is confirmed by user input, the contact between the desired tissue and the surface can be automatically maintained throughout at least a portion of the tissue treatment using visualization data of the end effector and surrounding tissue associated with the contact between the desired tissue and the instrument. The contact between the desired tissue and the surface can be automatically maintained, for example, by slight manipulation of the position, orientation, and / or FTC parameters of the end effector 4642.

[0256] If the surgical instrument 4600 is a handheld surgical instrument, the user can be provided with commands to control its position and / or orientation, for example, in the form of instructions that can be displayed on the display 4625 (Figure 22) of the surgical instrument 4600. Furthermore, the surgical instrument 4600 can issue an alert if user intervention is required to re-establish contact between the desired tissue and surface. On the other hand, non-user operations, such as manipulation of FTC parameters and / or joint angles, can be communicated, for example, from the surgical hub 2106 or the visualization system 2108 to the controller 4620 of the surgical instrument 4600. The controller 4620 can then cause the motor driver 4626 to implement the desired operation. If the surgical instrument 4600 is a surgical instrument connected to the robotic arm of the robotic system 2110, the position and / or orientation can be communicated, for example, from the surgical hub 2106 or the visualization system 2108 to the robotic system 2110.

[0257] Referring primarily to Figures 36A to 36C, the firing of a surgical instrument 4600 loaded with a first staple cartridge 4652 in a staple cartridge arrangement 4317 is illustrated. In the first stage, as shown in Figure 36A, the first landmark 4361 and the second landmark 4362 are superimposed on the surgical resection path 4312. The landmarks 4361 and 4362 are spaced apart by a distance (d1) defined by the size of the staple cartridge 4652 (e.g., 45), which represents the length of the staple line 4363 deployed on the surgical resection path 4312 by the staple cartridge 4652. A control circuit performing one or more embodiments of process 4300 can employ visualization data, as will be described in more detail elsewhere herein, to superimpose the landmarks 4361 and 4362 on the surgical resection path 4312 and continuously track and update their positions relative to predetermined important structures, such as anatomical structures 4364, 4365, 4366, and 4367.

[0258] As shown in Figure 36B, during firing, the staples of the staple line 4363 are deployed into the tissue, and the cutting member 4645 advances to cut the tissue along the surgical resection path 4312 between landmarks 4361 and 4362. In various examples, as the cutting member 4645 advances, the tissue being treated is stretched and / or displaced. Stretching and / or displacement of the tissue exceeding a predetermined threshold indicates that the cutting member 4645 is moving too quickly through the tissue being treated.

[0259] Figure 37 is a logical flowchart of process 4170, which shows a control program or logical configuration for adjusting the firing rate of a surgical instrument to address tissue stretching and / or displacement during firing. Process 4170 includes, in 4171, monitoring tissue stretching / displacement during firing of the surgical instrument, and, in 4172, if the tissue stretching / displacement is greater than a predetermined threshold, in 4173, adjusting the firing parameters.

[0260] One or more embodiments of process 4170 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4170 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4170. Additionally or alternatively, one or more embodiments of process 4170 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4170 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0261] In various examples, control circuits performing one or more aspects of process 4170 (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) in 4171 use visualization data from visualization systems (e.g., visualization systems 100, 160, 500, 2108) to monitor tissue stretching / displacement during firing of surgical instruments 4600. In the example shown in Figure 36B, in 4171, tissue stretching / displacement (d) is monitored using visualization data by tracking the distortion of the structured light grid projected onto the tissue during firing, and / or by tracking landmarks 4364, 4365, 4366, 4367 representing the locations of adjacent anatomical structures. Additionally, or alternatively, tissue stretching (d) can be monitored in 4171 by tracking the location of landmark 4362 during firing. In the example in Figure 36B, the tissue stretch / shatter (d) is the difference between the distance between landmarks 4361 and 4362 during firing (d1) and the distance between landmarks 4361 and 4362 during firing (d2). In any case, if the tissue stretch / shatter (d) is greater than or equal to a predetermined threshold in 4172, the control circuit in 4173 adjusts the firing parameters of the surgical instrument 4600 to reduce the tissue stretch / shatter (d). For example, the control circuit can instruct the controller 4620 to reduce the speed of the firing motor drive assembly 4604 by, for example, reducing the draw current of the firing motor 4602 and reducing the forward speed of the cutting member 4645. Additionally or alternatively, the control circuit can instruct the controller 4620 to pause the firing motor 4602 for a predetermined period of time to reduce the tissue stretch / shatter (d).

[0262] As shown in Figure 36C, after firing, the jaws of the end effector 4642 are released, and the stapled tissue shrinks due to the fired staples of the staple line 4363. Figure 36C shows the projected staple line length defined by distance (d1) and the actual staple line defined by a distance (d3) smaller than distance (d1). The difference between distances d1 and d2 represents the shrinkage / displacement distance (d').

[0263] Figure 38 is a logical flowchart of process 4180, which shows a control program or logical configuration for adjusting the proposed staple cartridge placement along the proposed surgical resection route. Process 4180 includes monitoring the contraction / displacement of the stapled tissue along the proposed surgical resection route after the firing of the staple cartridges in the proposed placement in 4081, and adjusting the proposed subsequent staple cartridge positions along the proposed surgical resection route.

[0264] One or more aspects of process 4180 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4180 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4180. Additionally or alternatively, one or more aspects of process 4180 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, process 4180 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0265] In various examples, control circuits (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that perform one or more aspects of process 4180 monitor the contraction / displacement of the stapled tissue along the proposed excision route 4312 in 4181. In the example shown in Figure 36C, the staple line 4363 is deployed from the staple cartridge of the staple cartridge arrangement 4317 into the tissue between landmarks 4361 and 4362. When the jaws of the end effector 4642 are released, the stapled tissue contracts / displaces by a distance (d'). The distance (d') is the difference between the distance (d1) between landmarks 4361 and 4362 before firing, which represents the length of the staple line 4361 proposed by arrangement 4317, and the distance (d3) which represents the actual length of the staple line 4363.

[0266] To avoid gaps between consecutive staple lines, the control circuit adjusts the position of subsequent staple cartridges in the proposed arrangement 4317 along the proposed surgical resection route 4312. For example, as shown in Figure 36C, the initially proposed staple line 4368 is removed and replaced by an updated staple line 4369 that extends through or covers a gap defined by distance (d'). In various embodiments, tissue shrinkage / shear (d') is monitored in 4181 by tracking the distortion of the structured light grid projected onto the tissue after the jaws of the end effector 4642 are unclamped, and / or by tracking landmarks 4364, 4365, 4366, 4367 representing the location of adjacent anatomical structures. Additionally, or alternatively, tissue shrinkage distance (d') can be monitored in 4181 by tracking the location of landmark 4362.

[0267] In various aspects, it may be desirable to use non-visible data from non-visible systems to corroborate visualized data derived from surgical visualization systems (e.g., visualization systems 100, 160, 500, 2108), and vice versa. For example, a non-visible system could include a ventilator configured to measure non-visible data such as the volume, pressure, partial pressure of carbon dioxide (PCO2), and partial pressure of oxygen (PO2) of the patient's lungs. Corroborating visualized data with non-visible data allows clinicians to provide visualized data derived from visualization systems with greater confidence in its accuracy. In addition, as will be discussed in detail below, corroborating visualized data with non-visible data allows clinicians to identify postoperative complications and determine the overall efficiency of organs. Corroboration can also be useful in cases of segmentectomy or complex lobectomy without hiatus.

[0268] In various contexts, clinicians may need to remove a portion of a patient's organ to eliminate vital structures such as tumors and / or other tissues. For example, the patient's organ might be the right lung. A clinician may need to remove a portion of the patient's right lung to eliminate diseased tissue. However, clinicians may not want to remove too much of the patient's lung surgically to ensure that lung function is not compromised too severely. Lung function can be assessed based on the maximum lung volume per breath, which represents the maximum lung capacity. In determining how much of the lung can be safely removed, clinicians must consider the limitations imposed by a predetermined reduction in maximum lung volume beyond which the lung loses its viability and requires complete organ resection.

[0269] In at least one example, the lung surface area and / or volume are estimated from visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). The lung surface area and / or volume can be estimated in terms of maximum lung volume or maximum lung volume per breath. In at least one example, the lung surface area and / or volume can be estimated at multiple points throughout the entire inspiratory / expiratory cycle. In at least one embodiment, before resecting a portion of the lung, visualization and non-visualization data can be used to correlate the lung surface area and / or volume determined by the visualization system with the lung volume determined by the ventilator. The correlation data can be used, for example, to construct a mathematical relationship between the lung surface area and / or volume, such as that derived from visualization data, and the lung volume determined by the ventilator. This relationship can be used to estimate the size of the lung portion that can be removed while maintaining a maximum lung volume reduction below a predetermined threshold that preserves the lung's viability.

[0270] Figure 39 shows a logical flowchart of a process 4750 for proposing the surgical resection of a portion of an organ, relating to at least one aspect of the present disclosure. Process 4750 is generally performed during surgery. Process 4750 includes, in 4752, proposing a portion of an organ to be resected based on visualization data from a surgical visualization system, wherein the resection of this portion is configured to result in an estimated reduction in the organ's volume. Process 4750 may further include, in 4754, determining a first value of the organ's non-visibility parameter before the resection of the portion, and in 4756, determining a second value of the organ's non-visibility parameter after the resection of the portion. Furthermore, in a particular example, process 4750 may further include, in 4758, supporting a predetermined volume reduction based on the first and second values ​​of the non-visibility parameter.

[0271] One or more embodiments of process 4750 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4750 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4750. Additionally or alternatively, one or more embodiments of process 4750 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, one or more embodiments of process 4750 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0272] In various embodiments, process 4750 can be implemented by a computer-implemented interactive surgical system 2100 (Figure 19) which includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include a remote server 2113. A control circuit that performs one or more embodiments of process 4750 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0273] Figure 41A shows a pair of lungs 4780 of a patient. In one embodiment, a clinician may use an imaging device 4782 to project a light pattern 4785, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4786 to enable determination of the topography or landscape of the surface of the patient's right lung 4786. The imaging device may be similar in various respects to imaging device 120 (Figure 1). As described elsewhere in this specification, a projected light array may be employed to determine the shape defined by the surface of the patient's right lung 4786 and / or the movement of the patient's right lung 4786 during surgery. In one embodiment, the imaging device 4782 may be coupled to a structured light source 152 of a control system 133. In one embodiment, as described elsewhere in this specification, a surgical visualization system, such as a surgical visualization system 100, may use the surface mapping logic 136 of a control circuit 133 to determine the topography or landscape of the surface of the patient's right lung 4786.

[0274] A clinician can provide a surgical system, such as surgical system 2100, with the type of procedure to be performed, such as a right upper lobectomy. In addition to providing the surgical procedure to be performed, the clinician can provide the surgical system with the maximum desired volume of organ to be removed during the surgery. Based on the visualization data acquired from imaging device 4782, the type of surgical procedure to be performed, and the maximum desired volume to be removed, the surgical system can propose a resection route 4788 for removing a portion of the right lung 4790 that satisfies all of the clinician's inputs. Other methods for proposing a surgical resection route are described elsewhere in this specification. To propose a resection route 4788, the surgical system can consider any number of additional parameters.

[0275] In various cases, it may be desirable to ensure that the volume of the resected organ results in a desired volume reduction from the patient's organs. To confirm that the resected volume results in a desired volume reduction, invisible data from an invisible system can be used. In one embodiment, the patient's maximum lung volume can be measured over time using a ventilator.

[0276] In at least one example, a clinician can use the surgical system 2100 in a surgical procedure to remove a lung tumor. The control circuit can identify the tumor from visualization data, as described above in relation to Figures 13A–13E, and can propose a surgical resection route to provide a safety margin around the tumor, as described above in relation to Figures 29–38. The control circuit can further estimate the lung volume at maximum lung capacity. Maximum lung capacity can be measured before surgery using a ventilator. The control circuit can estimate the reduction in maximum lung capacity associated with removing a portion of the lung, including the tumor and a safety margin of surrounding tissue, using a predetermined mathematical correlation between the visually estimated lung volume detected by the ventilator and the lung capacity. If the estimated reduction in lung capacity exceeds a predetermined safety threshold, the control circuit can warn the clinician and / or propose an alternative surgical resection route that reduces the lung capacity further.

[0277] Figure 41C shows a graph 4800 measuring the patient's maximum lung volume over time. Before partial resection of the organ (t1), the ventilator can measure the maximum lung volume. In Figure 41C, at time t1 before partial resection 4790, the maximum lung volume is measured to be 6L. In the above example, where the surgical procedure to be performed is a right upper lobectomy, the clinician may want to remove only the volume of the patient's lung that results in a predetermined volume reduction so as not to impair the patient's respiratory capacity. In one embodiment, the clinician may want to remove a portion that reduces the patient's maximum lung volume by up to approximately 17%, for example. Based on the surgical procedure and the desired volume reduction, the surgical system can propose a surgical resection route 4788 that achieves the removal of a portion of the lung while maintaining the maximum lung volume at a value of 83% or more of the unresected maximum lung volume.

[0278] As shown in Figure 41C, the clinician can use ventilator data to monitor the patient's maximum lung volume over time, including pre-resection 4802 and post-resection 4804 of lung portion 4790. At time t2, lung portion 4790 is resected along the proposed resection route 4788. As a result, the maximum lung volume measured by the ventilator decreases. The clinician can use ventilator data (pre-resection maximum lung volume 4802 and post-resection maximum lung volume 4804) to confirm that the post-resection lung volume results in the desired volume reduction from the lung. As shown in Figure 41C, post-resection, the maximum lung volume decreases to 5L, which is approximately a 17% decrease in maximum lung volume, roughly the same as the desired volume reduction. Using the ventilator data, the clinician can gain stronger confidence that the actual volume reduction is consistent with the desired volume reduction achieved by the proposed surgical resection route 4788. In other embodiments, if there is a discrepancy between the non-visualized data and the visualized data, such as a larger-than-expected decrease in maximum lung volume (too much lung resection) or a smaller-than-expected decrease in maximum lung volume (insufficient lung resection), the clinician can determine whether appropriate action is needed.

[0279] Referring here to Figure 41B, the right lung 4792 of a patient after partial 4790 resection is shown. After partial 4790 resection, clinicians may inadvertently cause air leakage 4794, resulting in air leaking into the space between the lung 4794 and the chest wall, which can cause pneumothorax 4796. As a result of air leakage 4794, the patient's maximum lung volume per breath steadily decreases over time as the right lung 4792 collapses. Air leakage can be detected by performing a dynamic surface area / volume analysis of the lung using visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and visually tracking changes in lung volume. Lung volume and / or surface area can be visually tracked at one or more points during the inspiratory / expiratory cycle to detect volume changes indicating air leakage 4794. In one embodiment, as discussed above, a projection light array from the imaging device 4782 can be used to monitor the movement of the patient's right lung 4786 over time, such as monitoring size reduction. In another embodiment, a surgical visualization system and surface mapping logic such as surface mapping logic 136 can be used to determine the topography or landscape of the surface of the patient's right lung 4786 and monitor changes in the topography or landscape over time.

[0280] In one embodiment, a clinician can use a non-visualizing system, such as a ventilator, to corroborate a volume reduction detected by a visualization system. Referring again to Figure 41C, as mentioned above, in 4802, the patient's maximum lung volume can be measured before and after partial lung resection to corroborate that the desired reduction in lung volume matches the actual reduction. In the above example where air leakage occurs inadvertently, in 4806, the maximum lung volume may steadily decrease over time. In one example, at time t2 immediately after partial resection, the clinician might notice that the maximum lung volume has decreased from 6L to 5L, which roughly coincides with a desired reduction in lung volume. After partial resection, the surgical visualization system can monitor the patient's lung volume over time. If the surgical visualization system determines that there is a change in volume, the clinician can measure the maximum lung volume again at time t3. At time t3, the clinician may notice a decrease in maximum lung volume from 5L to 4L, which supports data determined from the visualization system and suggests a possible air leak in the right lung 4792.

[0281] In addition, the control circuit can be configured to measure organ efficiency based on visualized and non-visible data. In one embodiment, organ efficiency can be determined by comparing visualized data with the difference in non-visible data before and after partial resection. In one example, the visualization system can generate a resection route that reduces maximum lung volume by 17%. The ventilator can be configured to measure maximum lung volume before and after partial resection. In the example shown in Figure 41C, the maximum lung volume is reduced by approximately 17% (from 6L to 5L). Since the reduction in actual lung volume (17%) to desired lung volume (17%) is almost 1:1, the clinician can determine that the lung is functionally efficient. In another example, the visualization system can generate a resection route that reduces maximum lung volume by 17%. However, the ventilator may measure a reduction in maximum lung volume greater than 17%, for example, 25%. In this example, since partial resection of the lung results in a greater-than-expected reduction in maximum lung volume, the clinician can determine that the lung is functionally inefficient.

[0282] Figure 40 shows a logical flowchart of process 4760 for estimating organ volume loss resulting from the removal of a selected portion of an organ, relating to at least one aspect of the present disclosure. Process 4760 is similar in many ways to the process in Figure 4750. However, unlike process 4750, process 4760 relies on a clinician selecting or proposing a surgical resection route when removing a portion of an organ during surgery. Process 4760 includes receiving input from the user in 4762 indicating the portion of the organ to be resected. Process 4760 further includes estimating the organ volume loss resulting from the removal of the portion in 4764. In at least one example, the organ is the patient's lung, and the volume loss estimated in 4762 is the reduction in the patient's maximum lung volume per breath. Visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108) can be used to estimate the volume loss corresponding to the removal of the portion. Process 4760 may further include, in 4766, determining a first value of the non-visible parameter of the organ before partial resection, and in 4768, determining a second value of the non-visible parameter of the organ after partial resection. Finally, process 4760 may further include, in 4768, supporting the estimated organ volume reduction based on the first and second values ​​of the non-visible parameter.

[0283] One or more embodiments of process 4760 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4760 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4760. Additionally or alternatively, one or more embodiments of process 4760 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, one or more embodiments of process 4760 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0284] In various embodiments, process 4760 can be implemented by a computer-implemented interactive surgical system 2100 (Figure 19) which includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include a remote server 2113. A control circuit that performs one or more embodiments of process 4760 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0285] In one example, a clinician may provide an input to a surgical visualization system, such as a surgical visualization system 2100, indicating the portion of an organ to be removed. In another example, the clinician may draw a resection route on a virtual 3D structure of the organ, such as a virtual 3D structure generated in 4104 during process 4100. In yet another example, the visualization system may overlay a treatment layout plan, which may take the form of a recommended treatment route, as described in more detail elsewhere in this specification. The recommended treatment route may be based on the type of surgery being performed. In one embodiment, the recommended treatment route may suggest various starting points and a variety of resection routes that the clinician can choose from, similar to the resection routes 4146, 4147, and 4148 described elsewhere in this specification. The proposed resection routes may be determined by the visualization system to avoid certain critical structures, such as arteries. The clinician may select one of the proposed resection routes until the desired resection route for removing a portion of the organ is completed.

[0286] In one example, a surgical visualization system can determine an estimated volume reduction of an organ based on a selected resection route. After resecting a predetermined portion along the resection route, the clinician may want to use non-visualized data to confirm, based on the visualized data, that the actual volume reduction is consistent with the estimated volume reduction. In one embodiment, this confirmation can be performed using a procedure similar to that described above for process 4750, where the organ is the lung. The clinician can measure the maximum lung volume before 4802 and after 4804 resection of the lung and compare the changes in maximum lung volume to determine the actual decrease in maximum lung volume. In one example, the surgical visualization system may estimate that the maximum lung volume will decrease by 17% based on the resection route proposed by the clinician. Before resection, the clinician may notice that the maximum lung volume is 6L (time t1). After resection, the clinician may notice that the maximum lung volume is 5L (time t2), which represents a decrease of approximately 17% in maximum lung volume. Clinicians can use this non-visualized data / ventilator data to gain stronger confidence that the actual volume loss is consistent with the estimated volume loss. In other cases, if there is a discrepancy between the non-visualized and visualized data, such as a larger-than-expected decrease in maximum lung volume (too much lung resection) or a smaller-than-expected decrease in maximum lung volume (insufficient lung resection), clinicians can determine whether appropriate action is needed.

[0287] In addition, the control circuit can be configured to measure organ efficiency based on visualized and non-visible data. In one embodiment, organ efficiency can be determined by comparing visualized data with the difference in non-visible data before and after partial resection. In one example, the surgical visualization system may estimate that the maximum lung volume will decrease by 17% based on the clinician's desired resection route. The ventilator can be configured to measure the maximum lung volume before and after partial resection. In the example shown in Figure 41C, the maximum lung volume decreases by approximately 17% (from 6L to 5L). Since the decrease in actual lung volume (17%) to the estimated lung volume (17%) is almost 1:1, the clinician can determine that the lung is functionally efficient. In another example, the surgical visualization system may estimate that the maximum lung volume will decrease by 17% based on the clinician's desired resection route. However, the ventilator may measure a decrease in maximum lung volume greater than 17%, for example, 25%. In this example, removing a portion of the lung results in a greater-than-expected decrease in maximum lung volume, allowing the clinician to determine that the lung is functionally inefficient.

[0288] As described above with respect to processes 4750 and 4760, clinicians can corroborate visualized data with non-visible data, such as by using a ventilator to measure maximum lung volume before and after partial lung resection. Another example of corroborating visualized data with non-visible data is through capnography.

[0289] Figure 42 shows graph 4810, which measures the partial pressure of carbon dioxide (PCO2) exhaled by the patient over time. In other examples, the partial pressure of oxygen (PO2) exhaled by the patient can be measured over time. Graph 4810 shows the PCO2 measured before resection 4812, immediately after resection 4814, and 1 minute after resection 4816. In Figure 42, before resection 4812, the PCO2 is measured to be approximately 40 mmHg (at time t1). In the above example, where the surgical procedure performed is a right upper lobectomy, the visualization system may hope or estimate a 17% reduction in lung volume. This desired or estimated volume reduction can be supported using the PCO2 level measured by the ventilator.

[0290] As shown in Figure 42, clinicians can use ventilator data to monitor the patient's PCO2 over time, such as before (4812) and after (4814) the resection of a portion of the lung (4790). At time t2, a portion of the lung (4790) is resected, and as a result, the PCO2 measured by the ventilator may decrease (4818). Clinicians can use ventilator data (pre-resection PCO2 4812 (t1) and post-resection PCO2 4814 (t2)) to ensure that the actual decrease in lung volume is consistent with the estimated or desired decrease in lung volume. As seen in Figure 42, immediately after partial resection, PCO2 decreased, which can be measured as a decrease of approximately 17% (approximately 33.2 mmHg) in PCO2. Using this non-visualized data / ventilator data, clinicians can have stronger confidence that the actual decrease in lung volume is consistent with the desired or estimated decrease in lung volume.

[0291] In other examples, clinicians can use non-visualized / PCO2 data to identify discrepancies when compared to visualized data. In one example, at time t2 4814 immediately after partial resection 4790, PCO2 was measured at 4820, which is higher than the PCO2 measured at 4812 before resection. The elevated PCO2 may result from inadvertent bronchial obstruction during surgery, leading to CO2 accumulation in the patient. In another example, at time t2 4814 immediately after partial resection 4790, PCO2 was measured at 4822, which is lower than the PCO2 measured at 4812 before resection and may be lower than expected. The decreased PCO2 may be due to inadvertent vascular obstruction during surgery, resulting in less O2 being delivered to the body and consequently less CO2 production. In either case, clinicians can take appropriate action to correct the situation.

[0292] Furthermore, changes in PCO2 can be measured at times other than immediately after resection (4814), for example, 1 minute after resection (4816, time t3). At time t3, other bodily functions (such as the liver) compensate for the change in PCO2 resulting from the resection. In this situation, PCO2 can be measured at approximately 40 mmHg, which is roughly the same as before resection (4812). The difference measured at time t3 between the pre-resection PCO2 (4812) and the current PCO2 may indicate the aforementioned inattentive obstruction. For example, at time t3, PCO2 may be measured at 4824, which is higher than the pre-resection PCO2 (4812), suggesting the possibility of inattentive bronchial obstruction, or PCO2 may be measured at 4826, which is lower than the pre-resection PCO2 (4812), suggesting the possibility of inattentive vascular obstruction.

[0293] In addition, the control circuit can be configured to measure organ efficiency based on visualized and non-visible data. In one embodiment, organ efficiency can be determined by comparing visualized data with the difference in non-visible data before and after partial resection. In one example, the surgical visualization system may estimate that lung volume will decrease by 17% based on the clinician's desired resection route. The ventilator can be configured to measure PCO2 before and after partial resection. In the embodiment shown in Figure 42, immediately after resection, PCO2 decreases by approximately 17%. Since the decrease in PCO2 (17%) is almost 1:1 with the estimated decrease in lung volume (17%), the clinician can determine that the lung is functionally efficient. In another example, the surgical visualization system may estimate that lung volume will decrease by 17% based on the clinician's desired resection route. However, the ventilator may measure a decrease in PCO2 greater than 17%, such as 25%. In this example, since partial resection of the lung causes a greater-than-expected decrease in PCO2, the clinician can determine that the lung is not functionally efficient.

[0294] In addition to the maximum lung volume and maximum PCO2 mentioned above, non-visible parameters including blood pressure or EKG data can be utilized. EKG data provides approximate frequency data regarding arterial deformation. This frequency data, where the surface geometry changes within a similar frequency range, can be useful in identifying important vascular structures.

[0295] As described above, it is sometimes desirable to use non-visualization data from a non-visualization system to support the visualization data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). In the above example, the non-visualization data is a means of supporting the visualization data after a portion of the organ has already been removed. In this example, it may be desirable to supplement the visualization data with non-visualization data before a portion of the organ is removed. In one example, non-visualization data can be used together with visualization data to help determine the characteristics of the organ to be operated on. In one embodiment, these characteristics may be abnormalities in the organ tissue that may be unsuitable for resection. Non-visualization and visualization data can be useful in informing the surgical visualization system and the clinician about areas to be avoided when planning the resection route of the organ. This can also be useful in the case of segmentectomy or complex lobectomy without hiats.

[0296] Figure 43 shows a logical flowchart of a process 4850 for detecting tissue abnormalities based on visualized and non-visible data, relating to at least one aspect of the present disclosure. Process 4850 is generally performed during surgical procedures. Process 4850 may include, in 4852, receiving first visualized data of an organ in a first state from a surgical visualization system, and in 4854, determining a first value for a non-visible parameter of the organ in the first state. Furthermore, process 4850 may include, in 4856, receiving second visualized data of an organ in a second state from the surgical visualization system, and in 4858, determining a second value for a non-visible parameter of the organ in the second state. The process may also include, in 4860, detecting tissue abnormalities based on the first visualized data, the second visualized data, the first value for the non-visible parameter, and the second value for the non-visible parameter.

[0297] One or more embodiments of process 4850 can be executed by one or more of the control circuits described herein (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more embodiments of process 4850 are executed by a control circuit (e.g., control circuit 400 in Figure 2A), which includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more embodiments of process 4850. Additionally or alternatively, one or more embodiments of process 4850 can be executed by combinational logic circuits (e.g., control circuit 410 in Figure 2B) and / or sequential logic circuits (e.g., control circuit 420 in Figure 2C). Furthermore, one or more embodiments of process 4850 can be executed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described herein.

[0298] In various embodiments, process 4850 can be implemented by a computer-implemented interactive surgical system 2100 (Figure 19) which includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include a remote server 2113. A control circuit that performs one or more embodiments of process 4850 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0299] Figure 44A shows the patient's right lung 4870 in a first state 4862. In one example, the first state 4862 may be a deflated state. In another example, the first state 4862 may be a collapsed state. An imaging device 4872 inserted through a cavity 4874 in the patient's chest wall 4876 is shown. The clinician can use the imaging device 4872 to project a light pattern 4882, such as stripes, grid lines, and / or dots, onto the surface of the right lung 4870 in 4880, to enable the determination of the topography or landscape of the surface of the patient's right lung 4870. The imaging device may be similar in various respects to imaging device 120 (Figure 1). As described elsewhere in this specification, a projected light array is employed to determine the shape defined by the surface of the patient's right lung 4870, and / or the movement of the patient's right lung 4870 during surgery. In one embodiment, the imaging device 4782 can be coupled to a structured light source 152 of a control system 133. In one embodiment, as described elsewhere in this specification, a surgical visualization system, such as the surgical visualization system 100, may use the surface mapping logic 136 of the control circuit 133 to determine the topography or landscape of the surface of the patient's right lung 4786. In the first state 4862 of the right lung 4870, parameters of the right lung 4870, such as the pressure (P1, or positive end-expiratory pressure (PEEP)) or the volume (V1), can be measured using a ventilator.

[0300] Figure 44B shows the patient's right lung 4870 in a second state 4864. In one example, the second state 4864 may be a partially inflated state. In another example, the second state 4864 may be a fully inflated state. The imaging device 4872 can be configured in 4880 to continuously emit a light pattern 4882 onto the surface of the lung 4870 to enable the determination of the topography or landscape of the surface of the patient's right lung 4870 in the second state 4864. In the second state 4864 of the right lung 4870, parameters of the right lung 4870 can be measured using a ventilator, such as the pressure in the second state (P2) which is greater than the pressure in the first state 4862 (P1), and the volume in the second state (V2) which is greater than the volume in the first state 4862 (V1).

[0301] Based on surface topography determined from the surgical visualization system and imaging device 4872, along with non-visual data (pressure / volume) determined from the ventilator, the surgical visualization system can be configured to determine tissue abnormalities in the right lung 4870. For example, in a first state 4862, the imaging device 4872 can determine the topography of the right lung 4870 in the first state 4662 (shown in Figure 44A, and shown in more detail in Figure 44C), and the ventilator can determine the pressure / volume of the first state. In a second state 4864, the imaging device 4872 can determine the topography of the right lung 4870 in the second state 4864 (shown in Figure 44D, and shown in more detail in Figure 44D), and the ventilator can determine the pressure / volume of the second state, which is greater than that of the first state due to the lung being partially or completely inflated. The visualization system can be configured to monitor changes in the topography of the right lung 4870 based on a known increase in pressure / volume, and in accordance with a known increase in pressure / volume. In one embodiment, these pressure / volume measurements from the ventilator can be correlated with surface deformation of the right lung 4870 to identify diseased areas within the lung and aid in staple placement.

[0302] In one embodiment, referring to Figures 44B and 44D, when the pressure increases from P1 to P2 (volume increases from V1 to V2), the surface topography determined from structured light 4880 changes compared to the first state 4862. In one example, the light pattern 4882 may be dots, and the dots are spaced apart by a distance(s) from each other as the lung size increases. In another example, the light pattern 4882 may be grid lines, and the grid lines are spaced apart or contoured as the lung size increases. Based on the known pressure increase, the imaging device can determine the region 4886 that did not change with the known pressure and volume increase. For example, if the imaging device 4872 emits a grid line and dot pattern 4882 onto the surface of the right lung 4870 (shown in Figures 6A-6D), the visualization system can be configured to monitor the contour of the grid lines and the relative position of the dots with respect to the known pressure / volume increase. If the visualization system indicates irregularities in the spacing of dots or the position and curvature of grid lines, the visualization system can determine that these areas correspond to potential tissue abnormalities such as subsurface voids 4886, or to areas where important structures 4884, such as tumors, may be located. In one embodiment, process 4100 refers to process 4100 that identifies anatomical structures of at least a portion 4105 of an anatomical organ related to a surgical operation, and process 4100 can identify abnormalities as described herein and superimpose these abnormalities onto a 3D structure.

[0303] In one example, a patient may have emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs (alveoli) of the lungs are damaged, and over time the inner walls of the air sacs weaken and rupture, creating large air spaces instead of many small air sacs. This reduces the internal surface area of ​​the lungs used for O2 / CO2 exchange, and therefore reduces the amount of oxygen that reaches the bloodstream. In addition, damaged alveoli do not function properly, trapping old air and leaving no room for fresh, oxygen-rich air. Since the air spaces in the lungs of emphysema patients represent areas of thin tissue, the outcome of stapling in those areas may be affected. As the tissue also weakens, the alveoli rupture as a result, and they are unable to hold staples passing through them as well.

[0304] When the lungs of an emphysema patient expand and contract, areas with subsurface cavities deform differently due to pressure changes compared to healthy tissue. Using the aforementioned process 4850, these weak tissue areas containing subsurface cavities can be detected, informing clinicians that stapling should be avoided through these areas, thereby reducing the likelihood of postoperative air leakage. The tissue deformation force of this process 4850 allows for the detection of these differences, enabling surgeons to guide stapler placement.

[0305] In the second case, the patient may have interstitial cancer. Prior to the procedure, the tumor may have been irradiated, damaging not only the tumor but also the surrounding tissue. Radiation often alters the properties of the tissue, making it harder and less compressible. If the surgeon needs to staple across this tissue, the change in tissue hardness should be considered when selecting the type of staple reload (for example, harder tissue will require staples formed higher).

[0306] When the lungs expand and contract, areas where the tissue is rigid deform differently compared to healthy tissue because the lung is less flexible. This tissue deformation force of process 4850 allows surgeons to detect these differences, enabling them to guide staple placement and cartridge / reload color selection.

[0307] In another embodiment, a memory such as memory 134 may be configured to store lung surface topography for known pressures and volumes. In this example, an imaging device such as imaging device 4872 can emit a light pattern to determine the topography of the patient's lung surface at a known first pressure or volume. A surgical system such as surgical system 2100 may be configured to compare a first determined topography at a known first pressure or volume with the topography stored in memory 134 for a given first pressure or volume. Based on this comparison, the visualization system may be configured to indicate potential tissue abnormalities in a single state. Focusing on these potentially abnormal areas, the visualization system can proceed to determine the topography of the patient's lung surface at a second known pressure or volume. The visualization system can compare the second determined surface topography with the topography stored in memory for a given second pressure or volume, as well as the topography determined at the known first pressure or volume. If the visualization system determines an anomaly potential region that overlaps with a first determined anomaly potential region, the visualization system can be configured to indicate the overlapping region as an anomaly potential with greater reliability based on a comparison of known first and second pressures or volumes.

[0308] In addition to the above, PO2 measurements from a ventilator allow for comparison of inflated lung volume (e.g., V2) and systolic lung volume (e.g., V1). Volume comparisons utilize EKG data to compare inspiration and expiration in a way that allows for comparison with blood oxygen concentration. This is then compared with anesthetic gas exchange measurements to determine respiratory volume versus oxygen consumption versus sedation level. Furthermore, EKG data provides approximate frequency data regarding arterial deformation. This frequency data, where surface geometry changes within a similar frequency range, can be useful in identifying important vascular structures.

[0309] In another embodiment, current tracking / procedure information can be compared to a preoperative planning simulation. For complex surgeries or high-risk procedures, clinicians can use preoperative patient scans to simulate the surgical approach. This dataset can be compared to real-time measurements on a display such as display 146, which may help enable surgeons to follow a specific preoperative plan based on training runs. This requires the ability to match reference landmarks between the preoperative scan / simulation and the current visualization. In one method, this may simply involve using object boundary tracking. Insights into how current device-tissue interactions compare to previous interactions (per patient) or predicted interactions (database or past patients) regarding tissue type distinctions, relative tissue deformation assessments, or subsurface structural differences can be stored in memory such as memory 134.

[0310] In one embodiment, the surface geometry may depend on the instrument position. If the change in surface geometry is not measured for each change in instrument position, a surface reference can be selected. When the instrument interacts with tissue and deforms the surface geometry, the change in surface geometry depending on the instrument position can be calculated by the surgical system. For a given change in instrument position upon contact with tissue, the change in tissue geometry may differ between areas containing subsurface structures such as critical structures 4884 and areas not containing such structures such as subsurface voids 4886. In one example, such as thoracic surgery, this may be present only in solid tissue, while it may be above the airway. The run-average of the change in instrument position and the change in surface geometry can be calculated by the surgical system for a given patient, taking into account a given patient-specific difference, using a surgical visualization system, or the value can be compared to a second set of previously collected data.

[0311] Exemplary clinical uses The various surgical visualization systems disclosed herein may be employed in one or more of the following clinical applications. These clinical applications are not exhaustive and are merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0312] The surgical visualization system can be employed in many different types of procedures in different specialties, such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, obesity / gastric surgery, and hepatobiliary and pancreatic surgery (HPB), as disclosed herein. For example, in urological surgery such as prostatectomy, the urinary tract may be detected in fat or connective tissue, and / or nerves may be detected, for example, in fat. For example, in gynecological oncology surgery such as hysterectomy, and colorectal surgery such as low anterior resection (LAR), the ureters may be detected, for example, in fat and / or connective tissue. For example, in thoracic surgery such as lobectomy, blood vessels may be detected in the lung or connective tissue, and / or nerves may be detected in connective tissue (e.g., esophagostomy). In bariatric surgery, blood vessels may be detected in fat. For example, in HPB surgeries such as hepatectomy or pancreatectomy, blood vessels may be detected in fat (extrahepatic) or connective tissue (extrahepatic), and bile ducts may be detected in parenchymal tissue (liver or pancreas).

[0313] In one example, a clinician may desire the removal of an endometrial fibroid. From a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may want to know during surgery which tissues constitute part of the intestine and which tissues constitute part of the rectum. In such cases, a surgical visualization system can display different types of tissue (intestine vs. rectum) as disclosed herein and transmit this information to the clinician via the imaging system. Furthermore, the imaging system can determine and communicate the proximity of the surgical device to the selected tissue. In such cases, the surgical visualization system can improve the efficiency of the procedure without serious complications.

[0314] In another example, a clinician (e.g., a gynecologist) may remain away from certain anatomical regions to avoid getting too close to vital structures, thus potentially preventing the removal of, for example, all of the endometriosis. A surgical visualization system, as disclosed herein, can enable a gynecologist to mitigate the risk of getting too close to vital structures so that surgical instruments can get close enough to remove all of the endometriosis, thereby improving the patient's outcome (democratized surgery). Such a system can enable a surgeon to "keep moving" during surgery, instead of repeatedly stopping and restarting to identify areas to avoid, for example, during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine arteries and ureters are vital structures, and the system may be particularly useful in hysterectomies and endometrial surgeries, given the presentation and / or thickness of the tissues involved.

[0315] In another example, clinicians may risk dissecting a vessel at a location that is too close, potentially affecting blood supply to lobes other than the target lobe. Furthermore, anatomical differences between patients may result in dissecting a vessel (e.g., a branching vessel) that affects different lobes depending on the individual patient. Surgical visualization systems, as disclosed herein, can enable the identification of the correct vessel at the desired location, thereby enabling clinicians to ensure they dissect the appropriate anatomical object. For example, the system can verify that the correct vessel is in the correct location, after which the clinician can safely dissect the vessel.

[0316] In another example, clinicians may make multiple incisions before finding the best location due to uncertainty in the anatomical structure of the blood vessels. However, since more incisions can increase the risk of bleeding, it is desirable to make the best location in the first step. Surgical visualization systems, as disclosed herein, can minimize the number of incisions by indicating the correct vessel and the best location for incision. For example, the ureters and cardinal ligaments are densely packed and present unique challenges during incision. In such cases, minimizing the number of incisions may be particularly desirable.

[0317] In another example, a clinician removing cancerous tissue (e.g., a surgical oncologist) may want to know the identification of key structures, the localization of the cancer, the staging of the cancer, and / or the assessment of the tissue's normality. Such information goes beyond what the clinician sees with the "naked eye." Surgical visualization systems, as disclosed herein, can determine and / or communicate such information to the clinician during surgery to enhance intraoperative decision-making and improve surgical outcomes. In certain cases, surgical visualization systems may be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches using either an endoscope or an exoscopy, for example.

[0318] In another example, a clinician (e.g., a surgical oncologist) might want to turn off one or more warnings regarding the approach of surgical instruments to one or more critical structures to avoid being overly cautious during surgery. In yet another example, a clinician might want to receive certain types of warnings, such as tactile feedback (e.g., vibration / buzzer), to indicate proximity and / or “no-fly zones” to ensure they remain sufficiently far from one or more critical structures. A surgical visualization system can provide adaptability, for example, based on the clinician’s experience and / or the desired aggressiveness of the procedure, as disclosed herein. In such an example, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. A surgical visualization system can assist in planning the next step(s) during surgery.

[0319] Various aspects of the subject matter described herein are illustrated in the following numbered examples. Example 1. A surgical system for use in surgical procedures, the surgical system comprising: a surgical visualization system; a control circuit configured to propose a portion of an organ to be resected based on visualization data from the surgical visualization system, wherein the resection of the portion is configured to result in an estimated reduction in the organ's volume; a first value of the non-visibility parameter of the organ before the resection of the portion; and a second value of the non-visibility parameter of the organ after the resection of the portion. Example 2. The surgical system according to Example 1, wherein the above-mentioned non-visible parameters include the maximum lung volume measured by a ventilator. Example 3. The surgical system according to Example 1, wherein the above-mentioned non-visible parameters include PCO2 measured by a ventilator. Example 4. The surgical system according to any one of Examples 1 to 3, wherein the control circuit is further configured to support the estimated capacity reduction based on the first value of the non-visible parameter and the second value of the non-visible parameter. Example 5. The surgical system according to Example 4, wherein the estimated volume reduction is below a predetermined threshold. Example 6. The surgical system according to any one of Examples 1 to 5, wherein the organs include the lungs. Example 7. The surgical system according to Example 6, wherein the control circuit is further configured to detect air leakage from the lung after the partial resection described above. Example 8. The surgical system according to Example 7, wherein the air leak is detected using dynamic visualization data of the lung from the visualization system after the above resection. Example 9. A surgical system for use in surgical procedures, the surgical system comprising: a surgical visualization system; and a control circuit configured to receive input from a user indicating a portion of an organ to be removed, and to estimate the reduction in the volume of the organ due to the removal of the portion based on visualization data from the surgical visualization system. Example 10. The surgical system according to Example 9, wherein the control circuit is further configured to determine a first value of the non-visibility parameter of the organ before partial resection. Example 11. The surgical system according to Example 10, wherein the control circuit is further configured to determine a second value of the non-visibility parameter of the organ after partial resection. Example 12. The surgical system according to Example 11, wherein the control circuit is further configured to support the estimated volume reduction of the organ based on the first value and the second value of the non-visible parameter. Example 13. The surgical system according to any one of Examples 10 to 12, wherein the above-mentioned non-visible parameters include the maximum lung volume measured by a ventilator. Example 14. The surgical system according to any one of Examples 10 to 12, wherein the above-mentioned non-visible parameters include PCO2 measured by a ventilator. Example 15. The surgical system according to any one of Examples 9 to 14, wherein the organs include the lungs. Example 16. The surgical system according to Example 15, wherein the control circuit is further configured to detect air leakage in the lung after resection using dynamic visualization data of the lung from the visualization system. Example 17. A surgical system for use in surgical procedures, the surgical system comprising: a surgical visualization system; and a control circuit configured to receive first visualization data of an organ in a first state from the surgical visualization system, determine a first value of the non-visibility parameter of the organ in the first state, receive second visualization data of the organ in a second state from the surgical visualization system, determine a second value of the non-visibility parameter of the organ in the second s...

Claims

1. A surgical system for use in a surgical procedure to remove a lung tumor, wherein the surgical system is: Surgical visualization system, Includes a control circuit, The control circuit identifies a lung tumor based on visualization data from the surgical visualization system. The control circuit proposes a surgical resection route to provide a safety margin around the lung tumor. The control circuit estimates at least one of the surface area and volume of the lung in an unresected state based on the visualization data from the surgical visualization system, and estimates a first maximum lung volume of the lung in an unresected state based on at least one of the estimated surface area and volume of the lung. The control circuit estimates the second maximum lung volume of the lung after a portion of the lung has been removed by the surgical resection route. The control circuit is a surgical system that warns the surgeon when the ratio of the second maximum lung volume to the first maximum lung volume is less than a predetermined value, or proposes a new surgical resection route that is estimated to achieve a maximum lung volume greater than the second maximum lung volume.

2. A surgical system for use in a surgical procedure to remove a lung tumor, wherein the surgical system is: Surgical visualization system, A ventilator that measures the actual first maximum lung capacity of a lung in which no parts have been removed, Includes a control circuit, The control circuit identifies a lung tumor based on visualization data from the surgical visualization system. The control circuit proposes a surgical resection route to provide a safety margin around the lung tumor. The control circuit estimates the second maximum lung volume of the lung after a portion of the lung has been removed by the surgical resection route. The control circuit is a surgical system that, when the ratio of the second maximum lung volume to the first maximum lung volume measured by the ventilator is less than a predetermined value, warns the surgeon or proposes a new surgical resection route that is estimated to achieve a maximum lung volume greater than the second maximum lung volume.

3. The actual maximum lung capacity or PCO of the lung in the state in which no portion has been removed. ２ In addition to measuring the actual maximum lung volume or PCO of the lung after the portion of the lung has been removed by the surgical resection route, ２ The surgical system according to claim 1, further comprising a ventilator for measuring [a specific value].

4. The surgical system according to claim 2, wherein the ventilator measures the actual maximum lung capacity of the lung after the portion of the lung has been removed by the surgical resection route.

5. The surgical system according to claim 1 or 2, wherein the estimation of the second maximum lung volume of the lung is performed by estimating at least one of the surface area and volume of the lung after a portion of the lung has been removed, based on the visualization data from the surgical visualization system, and estimating the second maximum lung volume of the lung after a portion of the lung has been removed, based on the estimated at least one of the surface area and volume of the lung after a portion of the lung has been removed.

6. The surgical system according to claim 1 or 2, wherein the control circuit is further configured to detect air leakage from the lung after the portion of the lung has been removed.

7. The surgical system according to claim 6, wherein the air leak is detected using dynamic visualization data of the lung from the surgical visualization system after the portion of the lung has been resected.

Images