The surgical system analyzes surgical trends and provides user recommendations.

Abstract

Various surgical systems configured to analyze surgical procedure trends are disclosed. The surgical system may be further configured to provide recommendations and / or adjust control algorithms executed by the surgical system according to identified trends. The identified trends may be utilized to determine baseline or recommended actions to be taken at decision points in the surgical procedure. The surgical system may be configured to provide pre-operative, intra-operative, or post-operative feedback to a user based on their decisions at the decision points relative to the determined baseline or recommended actions. A surgical control system communicatively connectable to a back-end computer system is provided, the surgical control system comprising: an imaging system comprising: an emitter configured to emit electromagnetic radiation (EMR), wherein at least a portion of the EMR is emitted as structured EMR; and an image sensor configured to receive EMR reflected from the surgical site; a control circuit coupled to the imaging system, the control circuit configured to: generate an image of the surgical site via reflected EMR received by the image sensor; determine a surgical operation being performed based on the image; receive a baseline surgical operation associated with the surgical operation from a back-end computer system; and provide a user recommendation according to a comparison between the surgical operation and the baseline surgical operation.

Description

[Background technology]

[0001] Surgical systems often incorporate imaging systems that allow clinicians to observe the surgical site and / or one or more parts thereof on one or more displays, such as monitors. The displays may be local to and / or remote from the operating room. The imaging system may include a camera-equipped scope that observes the surgical site and transmits images to a display viewable by the clinician. Examples of scopes include, but are not limited to, arthroscopy, angiography, bronchoscopy, cholangioscopy, colonoscopy, cytoscope, duodenoscopy, enteroscopy, esophagoduodenoscope (gastroscopy), endoscope, laryngoscope, nasopharyngolaryngoscope, sigmoidoscope, thoracoscopy, ureteroscope, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or communicated to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may be unrecognizable during surgery by certain imaging systems. In addition, certain imaging systems may be unable to communicate and / or transmit specific information to clinicians during surgery. [Overview of the project] [Means for solving the problem]

[0002] In one general embodiment, a surgical control system is disclosed that is communicably connectable to a backend computer system. The surgical control system comprises an imaging system and a control circuit coupled to the imaging system. The imaging system includes an emitter configured to emit electromagnetic radiation (EMR) and an image sensor configured to receive EMR reflected from the surgical site. At least a portion of the EMR is emitted as structured EMR. The control circuit is configured to generate an image of the surgical site via the reflected EMR received by the image sensor, to determine the surgical action being performed based on the image, to receive a baseline surgical action associated with the surgical action from the backend computer system, and to provide user recommendations according to a comparison between the surgical action and the baseline surgical action.

[0003] In another general embodiment, a computer system is disclosed that can communicately connect to a plurality of surgical hubs. Each of the plurality of surgical hubs can communicately connect to an imaging system. The computer system includes a control circuit configured to receive from the plurality of surgical hubs a plurality of images of a plurality of surgical sites acquired by each imaging system during a plurality of surgical procedures, and to determine a plurality of surgical outcomes, each of the plurality of surgical outcomes being associated with one of the surgical procedures, based on one of the plurality of images, to determine a baseline surgical action according to the plurality of images and the plurality of surgical outcomes, and to transmit the baseline surgical action to the plurality of surgical hubs.

[0004] In yet another general aspect, a method for controlling a surgical system communicably connectable to a backend computer system is disclosed. The control system includes an imaging system. The imaging system includes an emitter configured to emit electromagnetic radiation (EMR). At least a portion of the EMR is emitted as structured EMR, and an image sensor is configured to receive the EMR reflected from the surgical site. The method includes generating an image of the surgical site via the reflected EMR received by the image sensor, determining a surgical action being performed based on the image, receiving a baseline surgical action associated with the surgical action from the backend computer system, and providing a user recommendation according to a comparison between the surgical action and the baseline surgical action. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The novel features of the various aspects are set forth with particularity in the appended "Claims." However, the aspects described, as to both construction and method of operation, may best be understood in conjunction with the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 10 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify critical structures beneath the tissue surface. [Figure 2] FIG. 13 is a schematic diagram of a control system of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 2A] FIG. 16 shows a control circuit configured to control an aspect of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 2B] FIG. 19 shows a combinational logic circuit configured to control an aspect of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 2C] FIG. 22 shows a sequential logic circuit configured to control an aspect of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 3]Figure 1 is a schematic diagram illustrating the surgical apparatus, imaging apparatus, and triangulation between a vital structure and a vital structure for determining the depth dA of the vital structure beneath the tissue surface, according to at least one aspect of the present disclosure. [Figure 4] A schematic diagram of a surgical visualization system configured to identify important structures beneath the tissue surface according to at least one aspect of the present disclosure, the surgical visualization system including a pulsed light source for determining the depth dA of the important structures beneath the tissue surface. [Figure 5] This is a schematic diagram of a surgical visualization system, including an imaging device and a surgical device, according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify important structures beneath the tissue surface. [Figure 6] A schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, the surgical visualization system being configured to identify important structures embedded in tissue. [Figure 7A] Figure 7A shows an image of an important structure taken with the three-dimensional camera shown in Figure 6, according to at least one aspect of the present disclosure, where Figure 7A is an image from the left lens of the three-dimensional camera and Figure 7B is an image from the right lens of the three-dimensional camera. [Figure 7B] Figure 7A shows an image of an important structure taken with the three-dimensional camera shown in Figure 6, according to at least one aspect of the present disclosure, where Figure 7A is an image from the left lens of the three-dimensional camera and Figure 7B is an image from the right lens of the three-dimensional camera. [Figure 8] Figure 6 is a schematic diagram of a surgical visualization system according to at least one aspect of this disclosure, which can determine the camera-to-critical structure distance dw from a three-dimensional camera to a critical structure. [Figure 9] This is a schematic diagram of a surgical visualization system that utilizes two cameras to determine the location of a buried critical structure, according to at least one aspect of the present disclosure. [Figure 10A] This is a schematic diagram of a surgical visualization system, according to at least one aspect of the present disclosure, which utilizes a camera that moves axially between a plurality of known locations to determine the location of a buried critical structure. [Figure 10B] Figure 10A is a schematic diagram of a surgical visualization system according to at least one aspect of the present disclosure, in which a camera is moved axially and rotationally between a plurality of known locations to determine the location of a buried critical structure. [Figure 11] This is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure. [Figure 12] This is a schematic diagram of a structural light source for a surgical visualization system according to at least one aspect of the present disclosure. [Figure 13A] This is a graph of absorption coefficient versus wavelength for various biomaterials according to at least one aspect of the present disclosure. [Figure 13B] This is a schematic diagram of visualization of an anatomical structure via a spectral surgical visualization system according to at least one aspect of the present disclosure. [Figure 13C] Figure 13C depicts exemplary hyperspectral identification signatures for distinguishing anatomical structures from camouflage, according to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus camouflage, Figure 13D is a graph representation of an arterial signature versus camouflage, and Figure 13E is a graph representation of a nerve signature versus camouflage. [Figure 13D] Figure 13C depicts exemplary hyperspectral identification signatures for distinguishing anatomical structures from camouflage, according to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus camouflage, Figure 13D is a graph representation of an arterial signature versus camouflage, and Figure 13E is a graph representation of a nerve signature versus camouflage. [Figure 13E] Figure 13C depicts exemplary hyperspectral identification signatures for distinguishing anatomical structures from camouflage, according to at least one aspect of the present disclosure, where Figure 13C is a graph representation of a ureteral signature versus camouflage, Figure 13D is a graph representation of an arterial signature versus camouflage, and Figure 13E is a graph representation of a nerve signature versus camouflage. [Figure 14]A schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense the distance to an important anatomical structure, according to at least one aspect of the present disclosure, the time-of-flight measurement system includes a transmitter (emitter) and a receiver (sensor) positioned on a common device. [Figure 15] Figure 17A is a schematic diagram of the radiated wave, received wave, and delay between the radiated wave and the received wave of the NIR time-of-flight measurement system according to at least one aspect of the present disclosure. [Figure 16] The present disclosure describes at least one aspect of a NIR time-of-flight measurement system configured to sense the distance to different structures, the time-of-flight measurement system comprising a transmitter (emitter) and a receiver (sensor) on separate devices. [Figure 17] This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 18] A surgical system used to perform surgical procedures in an operating room, according to at least one aspect of this disclosure. [Figure 19] This figure shows a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 20] This is a diagram of a situational awareness surgical system according to at least one aspect of the present disclosure. [Figure 21] This figure shows a timeline depicting the hub's situational awareness according to at least one aspect of this disclosure. [Figure 22] This is a block diagram of a surgical system according to at least one aspect of the present disclosure. [Figure 23] This is a logical flowchart of a process for providing dynamic surgical recommendations to a user, according to at least one aspect of this disclosure. [Figure 24] A surgical visualization showing recommended surgical instrument positioning, according to at least one aspect of this disclosure. [Modes for carrying out the invention]

[0006] The applicant of this application also owns the following U.S. patent applications filed concurrently, each of which is incorporated herein by reference in its entirety: • Agent reference number END9228USNP1 / 190580-1M, title "STRUCTURED MULTI SPECTRAL COMPUTATIONAL ANALYSIS", • Agent reference number END9227USNP1 / 190579-1, title "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM", Agent reference number END9226USNP1 / 190578-1, title "SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS", Agent reference number END9225USNP1 / 190577-1, title "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS", Agent reference number END9224USNP1 / 190576-1, title "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS", • Agent reference number END9223USNP1 / 190575-1, title "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA", Agent reference number END9222USNP1 / 190574-1, title "SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED", Agent reference number END9221USNP1 / 190573-1, title "SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN", Agent reference number END9220USNP1 / 190572-1, title "SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS", Agent reference number END9219USNP1 / 190571-1, title "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE", Agent reference number END9218USNP1 / 190570-1, title "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT", Patent attorney reference number END9217USNP1 / 190569-1, title "DYNAMIC SURGICAL VISUALIZATION SYSTEMS".

[0007] The applicant of this application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety: U.S. Patent Application No. 16 / 354,417, title "INPUT CONTROLS FOR ROBOTIC SURGERY", U.S. Patent Application No. 16 / 354,420, title "DUAL MODE CONTROLS FOR ROBOTIC SURGERY", U.S. Patent Application No. 16 / 354,422, title "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY", U.S. Patent Application No. 16 / 354,440, titled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY", U.S. Patent Application No. 16 / 354,444, titled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE" U.S. Patent Application No. 16 / 354,454, titled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS", U.S. Patent Application No. 16 / 354,461, title "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS", U.S. Patent Application No. 16 / 354,470, titled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS", U.S. Patent Application No. 16 / 354,474, titled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES", • U.S. Patent Application No. 16 / 354, 478, titled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK", and U.S. Patent Application No. 16 / 354,481, titled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS".

[0008] Furthermore, the applicant of this application owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety: U.S. Patent Application No. 16 / 128,179, title "SURGICAL VISUALIZATION PLATFORM", U.S. Patent Application No. 16 / 128,180, title "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE", U.S. Patent Application No. 16 / 128,198, title "SINGULAR EMR SOURCE EMITTER ASSEMBLY", U.S. Patent Application No. 16 / 128,207, title "COMBINATION EMITTER AND CAMERA ASSEMBLY", U.S. Patent Application No. 16 / 128,176, title "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES", U.S. Patent Application No. 16 / 128,187, title "SURGICAL VISUALIZATION OF MULTIPLE TARGETS", U.S. Patent Application No. 16 / 128,192, title "VISUALIZATION OF SURGICAL DEVICES", U.S. Patent Application No. 16 / 128,163, title "OPERATIVE COMMUNICATION OF LIGHT", U.S. Patent Application No. 16 / 128,197, title "ROBOTIC LIGHT PROJECTION TOOLS", • U.S. Patent Application No. 16 / 128,164, title "SURGICAL VISUALIZATION FEEDBACK SYSTEM", U.S. Patent Application No. 16 / 128,193, title "SURGICAL VISUALIZATION AND MONITORING", • U.S. Patent Application No. 16 / 128,195, title "INTEGRATION OF IMAGING DATA", • U.S. Patent Application No. 16 / 128,170, title "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS", • U.S. Patent Application No. 16 / 128,183, title "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS", • U.S. Patent Application No. 16 / 128,172, titled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER", and U.S. Patent Application No. 16 / 128,185, title "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION".

[0009] The applicant of this application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety: • U.S. Patent Application No. 15 / 940,627, title "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", current U.S. Patent Publication No. 2019 / 0201111. • U.S. Patent Application No. 15 / 940,676, titled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", current U.S. Patent Publication No. 2019 / 0201142. • U.S. Patent Application No. 15 / 940,711, titled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", current Patent Application Publication No. 2019 / 0201120, and U.S. Patent Application No. 15 / 940,722, titled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY," current publication No. 2019 / 0200905.

[0010] The applicant of this application owns the following U.S. patent applications filed on December 4, 2018, the disclosures of each of these are incorporated herein by reference in their entirety: • U.S. Patent Application No. 16 / 209,395, title "METHOD OF HUB COMMUNICATION", current U.S. Patent Publication No. 2019 / 0201136. • U.S. Patent Application No. 16 / 209,403, title "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", current U.S. Patent Publication No. 2019 / 0206569. U.S. Patent Application No. 16 / 209,407, title "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", current U.S. Patent Publication No. 2019 / 0201137. • U.S. Patent Application No. 16 / 209,416, title "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS", current U.S. Patent Publication No. 2019 / 0206562. U.S. Patent Application No. 16 / 209,423, title "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", current U.S. Patent Publication No. 2019 / 0200981. U.S. Patent Application No. 16 / 209,427, titled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES," current U.S. Patent Publication No. 2019 / 0208641. U.S. Patent Application No. 16 / 209,433, title "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB", current U.S. Patent Application Publication No. 2019 / 0201594. • U.S. Patent Application No. 16 / 209,447, title "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB", current U.S. Patent Publication No. 2019 / 0201045. • U.S. Patent Application No. 16 / 209,453, title "METHOD FOR CONTROLLING SMART ENERGY DEVICES", current U.S. Patent Publication No. 2019 / 0201046. • U.S. Patent Application No. 16 / 209,458, title "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE", current U.S. Patent Publication No. 2019 / 0201047. • U.S. Patent Application No. 16 / 209,465, title "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION", current U.S. Patent Publication No. 2019 / 0206563. U.S. Patent Application No. 16 / 209,478, titled "Method for Situational Awarenness for Surgical Network or Surgical Network Connected Device Capable of Adjusting Function Based on a Sensed Situation or Usage," current U.S. Patent Publication No. 2019 / 0104919. • U.S. Patent Application No. 16 / 209,490, titled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION", current U.S. Patent Publication No. 2019 / 0206564, and U.S. Patent Application No. 16 / 209,491, titled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS," current U.S. Patent Application Publication No. 2019 / 0200998.

[0011] Before describing in detail the various embodiments of the surgical visualization platform, it should be noted that the exemplary embodiments are not limited in application or use to the structural and arrangement details of the components illustrated in the accompanying drawings and specification. The exemplary embodiments may be implemented or incorporated into other embodiments, variations, and modifications, and may be implemented or performed in various ways. Furthermore, unless otherwise specified, the terms and expressions used herein have been selected for the purpose of describing the exemplary embodiments for the convenience of the reader and are not intended to limit them. Furthermore, it should be understood that one or more embodiments, expressions of embodiments, and / or embodiments described below may be combined with any one or more other embodiments, expressions of embodiments, and / or embodiments described below.

[0012] Surgical visualization system This disclosure pertains to a surgical visualization platform that leverages “digital surgery” to obtain additional information about a patient’s anatomical structure and / or surgical procedure. The surgical visualization platform is further configured to communicate the data and / or information to one or more clinicians in a useful manner. For example, various aspects of this disclosure provide improved visualization of a patient’s anatomical structure and / or surgical procedure.

[0013] "Digital surgery" may encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, and connectivity both inside and outside the operating room (OR). The various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, but are not limited to use with robotic surgical systems. In certain examples, advanced surgical visualization can be performed without a robot, and / or with limited and / or optional robot assistance. Similarly, digital surgery can be performed without a robot, and / or with limited and / or optional robot assistance.

[0014] In certain cases, surgical systems incorporating a surgical visualization platform may enable smart incisions to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors, among other anatomical structures. In other cases, critical structures may be foreign body structures in the anatomical field, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, and / or plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Exemplary critical structures are further described herein. Smart incisions can provide improved intraoperative guidance for incisions and / or enable smarter determination, for example, through critical anatomical detection and avoidance techniques.

[0015] Surgical systems incorporating surgical visualization platforms can also enable smart anastomosis, providing more reliable anastomoses at optimal locations through improved workflows. Cancer localization methods can also be improved by the various surgical visualization platforms and procedures described herein. For example, cancer localization methods can identify and track the location, orientation, and margins of a tumor. In certain cases, cancer localization methods can compensate for the movement of instruments, patients, and / or patient anatomical structures during surgical procedures, and reguide the clinician to the target point.

[0016] In certain aspects of this disclosure, the surgical visualization platform may provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize the type and health of tissue without requiring physical touch, particularly during incision and / or placement of staple devices within the tissue. Certain tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can, for example, preoperatively locate, map, and ideally diagnose lymphatic systems and / or lymph nodes involved in cancer diagnosis and staging.

[0017] During surgical procedures, the information available to clinicians via the naked eye and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures, such as those embedded in or covered by organs, may be at least partially hidden from view, i.e., invisible. In addition, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or difficult to grasp with the naked eye. Furthermore, certain structures may move before surgery (e.g., before the surgical procedure but after the preoperative scan) and / or during surgery. In such cases, clinicians may not be able to accurately determine the location of critical structures during surgery.

[0018] When the location of critical structures is uncertain, and / or the proximity of critical structures to surgical tools is unknown, the clinician's decision-making process can be hindered. For example, a clinician may avoid certain areas to avoid inadvertently cutting critical structures. However, the avoided areas may be unnecessarily large and / or at least partially in the wrong location. Due to uncertainty and / or excessive caution, clinicians may not access certain desired areas. For example, even if critical structures are not in that particular area, and / or if the clinician's actions in that area would have no adverse effects, excessive caution may cause a clinician to try to avoid critical structures, leaving behind parts of tumors and / or other undesirable tissue. In certain cases, increased knowledge and / or certainty can improve surgical outcomes, which in turn allows surgeons to be more precise and, in certain cases, more cautious / more aggressive with respect to certain anatomical areas.

[0019] In various embodiments, the Disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one embodiment, the Disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various embodiments, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what the clinician sees with the naked eye and / or what the imaging system can perceive and / or communicate to the clinician. Various surgical visualization systems can improve outcomes in various cases by reinforcing and enhancing what the clinician can know before tissue handling (e.g., incision).

[0020] For example, a visualization system may include a first optical emitter configured to emit multiple spectral waves, a second optical emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to spectral waves (spectroscopic imaging), and / or the light pattern. Note that throughout the following disclosure, any reference to “light” may include electromagnetic radiation (EMR) or photons of the visible and / or invisible portions of the EMR wavelength spectrum, unless otherwise specified with respect to visible light. A surgical visualization system may also include an imaging system and a receiver and a control circuit that signals to the imaging system. Based on the output from the receiver, the control circuit may determine a geometric surface map of the visible surface at the surgical site, i.e., a three-dimensional surface topography, and one or more distances to the surgical site. In certain examples, the control circuit may determine one or more distances to at least partially hidden structures. Furthermore, the imaging system may communicate the geometric surface map and one or more distances to the clinician. In such examples, the magnified view of the surgical site provided to the clinician can provide a diagram of structures hidden within the background related to the surgical site. For example, an imaging system can virtually magnify hidden structures onto a geometric surface map of the hidden and / or obstructing tissue, similar to lines drawn on the ground to indicate subsurface utility piping. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to the visible and obstructing tissue and / or to structures that are at least partially hidden, as well as the depth of structures hidden beneath the visible surface of the obstructing tissue. For example, a visualization system can determine the distance to a magnified line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of this disclosure, surgical visualization systems for intraoperative identification and avoidance of critical structures are disclosed. Such surgical visualization systems can provide clinicians with useful information during surgical procedures. As a result, knowing that the surgical visualization system is tracking critical structures that may be approached during incision, for example, a ureter, specific nerves, and / or important blood vessels, clinicians can maintain confidence and momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide clinicians with sufficient time to pause and / or slow down the surgical procedure and assess the proximity to critical structures to prevent inadvertent damage to those structures. The surgical visualization system can provide clinicians with an ideal, optimized, and / or customizable amount of information, enabling them to move through tissues confidently and / or quickly while avoiding inadvertent damage to healthy tissue and / or critical structures, thereby minimizing the risk of damage resulting from the surgical procedure.

[0022] Figure 1 is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can produce a visual representation of critical structures 101 within the anatomical surgical field. The surgical visualization system 100 may be used, for example, for clinical analysis and / or medical intervention. In certain examples, the surgical visualization system 100 can be used during surgery to provide clinicians with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during surgical procedures. The surgical visualization system 100 is configured for intraoperative identification of critical structures and / or to facilitate avoidance of critical structures 101 by surgical instruments. For example, by identifying critical structures 101, clinicians can avoid operating surgical instruments around critical structures 101 and / or a given proximal area of ​​critical structures 101 during surgical procedures. Clinicians can avoid incisions around, for example, veins, arteries, nerves, and / or blood vessels, which are identified as critical structures 101. In various cases, critical structure 101 may be determined on a patient-by-patient and / or treatment-by-treatment basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. The combined features of the surgical visualization system 100 allow for the determination of the location of critical structures 101 within the anatomical surgical field, and / or the proximity of surgical instruments 102 to the surface of visible tissue 105 and / or critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system, including, for example, an imaging device 120 such as a camera configured to provide real-time images of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate spectral cubes of images based on molecular responses to different wavelengths. Images from the imaging device 120 can be provided to a clinician, and in various aspects of this disclosure, the images can be augmented with additional information based on tissue identification, context mapping, and the distance sensor system 104. In such an example, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems can work together to provide clinicians with advanced data synthesis and integrated information during surgery.

[0024] The imaging device may include, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structural light patterns (visible or invisible). In various embodiments of this disclosure, the imaging system may include, for example, an imaging device such as an endoscope. Additionally or alternatively, the imaging system may include, for example, an imaging device such as an arthroscope, angioscope, bronchoscope, cholangioscope, coloscope, cystoscope, duodenoscope, intestinaloscope, esophagogastroduodenoscope (gastroscopy), laryngoscope, nasopharyngo-neproscope, sigmoidoscopy, thoracoscopy, ureteroscope, or exoscopy. In other examples, such as in open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of this disclosure, the tissue identification subsystem can be achieved with a spectral imaging system. The spectral imaging system may rely, for example, on hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, issued March 1, 2016, entitled "SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING," which is incorporated herein by reference in its entirety.

[0026] In various aspects of this disclosure, the surface mapping subsystem can be achieved with an optical pattern system, as further described herein. The use of optical patterns (or structural light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structural illuminance is the process of projecting a known pattern (often a grid or horizontal bars) onto a surface. U.S. Patent Application Publication 2017 / 0055819, “SET COMPRISING A SURGICAL INSTRUMENT,” published March 2, 2017, and U.S. Patent Application Publication 2017 / 0251900, “DEPICTION SYSTEM,” published September 7, 2017, disclose a surgical system comprising a light source and a projector for projecting a light pattern. Both publications, “SET COMPRISING A SURGICAL INSTRUMENT,” and “DEPICTION SYSTEM,” are incorporated herein by reference in their entirety.

[0028] In various aspects of this disclosure, the distance determination system can be incorporated into a surface mapping system. For example, structural light can be used to generate a three-dimensional virtual model of a visible surface and determine various distances to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to identified tissues (or other structures) at the surgical site.

[0029] Figure 2 is a schematic diagram of a control system 133 that may be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 that communicates with a memory 134. The memory 134 stores instructions that can be executed by the control circuit 132 to determine and / or recognize critical structures (e.g., critical structure 101 in Figure 1), determine and / or calculate one or more distances and / or three-dimensional digital displays, and communicate specific information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (such as the imaging device 120 in Figure 1), one or more displays 146, or one or more control units 148, or any combination of these elements. The camera 144 may include one or more image sensors 135 (e.g., visible light, spectral imaging device, three-dimensional lens) for receiving signals from various light sources emitting light in various visible and invisible spectra. The display 146 may include one or more screens or monitors for displaying real, virtual, and / or virtually augmented images and / or information to one or more clinicians.

[0030] In various embodiments, the central component of the camera 144 is the image sensor 135. Generally, modern image sensors 135 are solid-state electronic devices containing up to several million distinct light-detecting regions called pixels. Image sensor 135 technology is classified into one of two categories: charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) imaging devices, with short-wave infrared (SWIR) being a more recent development in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (marketed as "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 have sensitivity to wavelengths of approximately 350–1050 nm, although this range is typically given as 400–1000 nm. CMOS sensors are generally more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Therefore, there are two types of color CCD cameras: one-chip and three-chip. One-chip color CCD cameras provide a common, low-cost imaging solution, using mosaic (e.g., Bayer) optical filters to separate the input light into a set of colors and resolving a full-color image using an interpolation algorithm. Each color is then directed to a different set of pixels. Three-chip color CCD cameras provide higher resolution by using prisms, directing each section of the incident spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a distinct RGB intensity value rather than using an algorithm to determine the color. Three-chip cameras offer very high resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structural light source 152. In certain examples, a single light source can pulse wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structural light source 152. Alternatively, a single light source can pulse wavelengths of light in the visible spectrum (e.g., infrared spectral light) and on the visible spectrum. The spectral light source 150 may be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various examples, tissue identification logic 140 can identify important structures via data from the spectral light source 150 received by the image sensor portion 135 of the camera 144. Surface mapping logic 136 can determine the surface contour of visible tissue based on reflected structural light. By time-of-flight measurement, distance determination logic 141 can determine one or more distances to visible tissue and / or important structures 101. One or more outputs from the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 may be provided to the imaging logic 138 and combined, integrated, and / or superimposed so as to be communicated to the clinician via the display 146 of the imaging system 142.

[0032] Herein, with reference to Figures 2A to 2C, various embodiments of the control circuit 132 for controlling various embodiments of the surgical visualization system 100 are briefly described. Figure 2A shows a control circuit 400 configured to control an embodiment of the surgical visualization system 100 according to at least one embodiment of the present disclosure. The control circuit 400 can be configured to implement various processes described herein. The control circuit 400 may include a microcontroller comprising one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions, which, when executed by the processor 402, cause the processor 402 to execute machine instructions for implementing various processes described herein. The processor 402 may be either a single-core processor or a multi-core processor known in the art. The memory circuit 404 may comprise volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0033] Figure 2B shows a combinational logic circuit 410 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 may comprise a finite state machine comprising a combinational logic 412 configured to receive data associated with a surgical instrument or tool at an input 414, process the data by combinational logic 412, and provide an output 416.

[0034] Figure 2C shows a sequential logic circuit 420 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, a combinational logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store the current state of the finite state machine. In certain examples, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic 422 is configured to receive data associated with a surgical instrument or system from an input 426, process the data by the combinational logic 422, and provide an output 428. In other aspects, the circuit may include a combination of a processor (e.g., a processor 402 in Figure 2A) and a finite state machine that implements various processes described herein. In other embodiments, a finite state machine may include a combination of combinational logic circuits (e.g., the combinational logic circuit 410 in Figure 2B) and sequential logic circuits 420.

[0035] Referring again to the surgical visualization system 100 in Figure 1, the critical structure 101 may be the target anatomical structure. For example, the critical structure 101 may be, among other anatomical structures, arteries such as the ureter and superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other examples, the critical structure 101 may be a foreign body structure in the anatomical surgical field, such as a surgical device, surgical fastener, clip, clasp, bougie, band, and / or plate. Exemplary critical structures are described herein and further in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,192 entitled "VISUALIZATION OF SURGICAL DEVICES," filed September 11, 2018, each of which is incorporated herein by reference in whole.

[0036] In one embodiment, the important structure 101 may be embedded within the tissue 103. In other words, the important structure 101 may be located beneath the surface 105 of the tissue 103. In such an example, the tissue 103 obscures the important structure 101 from the clinician's view. The important structure 101 is also obscured from the view of the imaging device 120 by the tissue 103. The tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other examples, the important structure 101 may be partially obscured from the view.

[0037] Figure 1 also depicts a surgical instrument 102. The surgical instrument 102 includes an end effector having opposing jaws extending from the distal end of the shaft of the surgical instrument 102. The surgical instrument 102 may be any suitable surgical instrument, such as an incision instrument, stapler, gripping instrument, clip applicator, and / or an energy device including a unipolar probe, bipolar probe, ablation probe, and / or an ultrasonic end effector. Additionally or alternatively, the surgical instrument 102 may include another imaging or diagnostic mode, such as an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 may be configured to achieve identification of one or more critical structures 101 and proximity of the surgical instrument 102 to the critical structures 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as visible light, spectral light waves (visible or invisible), and structural light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral image sensor, detector, and / or three-dimensional camera lens). For example, the imaging device 120 may include a right-hand and left-hand lens used together to generate a three-dimensional image of the surgical site by simultaneously recording two two-dimensional images, to render a three-dimensional image of the surgical site, and / or to determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images showing topography of visible tissue, as well as the identification and location of hidden important structures, as further described herein. For example, as shown in Figure 1, the field of view of the imaging device 120 can be superimposed on the light pattern (structural light) on the surface 105 of the tissue.

[0039] In one embodiment, the surgical visualization system 100 may be incorporated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112 and 114 include rigid structural members 116 and joints 118 that can include servo motor control. The first robotic arm 112 is configured to operate a surgical instrument 102, and the second robotic arm 114 is configured to operate an imaging device 120. A robotic control unit may be configured to generate control movements to the robotic arms 112 and 114 that can act on the surgical instrument 102 and the imaging device 120, for example.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit light patterns such as stripes, grid lines, and / or dots to enable determination of the topography or condition of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 may be emitted from an emitter 106 located, for example, on the surgical apparatus 102 and / or one of the robotic arms 112, 114 and / or the imaging device 120. In one embodiment, the projected light array 130 is used to determine the shape defined during surgery by the surface 105 of the tissue 103 and / or the movement of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0041] In one embodiment, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach critical structures 101. The imaging device 120 and the optical waveform emitter 123 on it may be positionable by a robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable the identification of types of anatomical structures and / or physical structures, such as critical structures 101. Identification of critical structures 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one embodiment, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 may include, for example, a photoacoustic imaging system. In other examples, the optical waveform emitter 123 may be located on a surgical device separate from the imaging device 120.

[0042] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one embodiment, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including an emitter such as an emitter 106 and a receiver 108, which may be located on the surgical apparatus 102. In other examples, the time-of-flight emitter may be separate from the structural light emitter. In one common embodiment, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a matched sensor. The time-of-flight distance sensor system 104 can detect "time of flight," that is, the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, the distance sensor system 104 can determine the distance to the surface 105 of the tissue 103 immediately in front of the distance sensor system 104. Furthermore, referring to Figure 1, d e d is the emitter-tissue distance from emitter 106 to surface 105 of tissue 103, and t d is the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. Using the distance sensor system 104, the emitter-tissue distance d is measured. e It is possible to determine the distance between the device and the tissue, d. t This can be obtained from the known position of the emitter 106 on the shaft of the surgical instrument 102 relative to the distal end of the surgical instrument 102. In other words, if the distance between the emitter 106 and the distal end of the surgical instrument 102 is known, then the instrument-tissue distance d t is the emitter-tissue distance d eThis can be determined from the following. In certain examples, the shaft of the surgical device 102 may include one or more articulated joints and may be articulate with respect to the emitter 106 and jaws. The articulated configuration may include, for example, a multi-articulated vertebral-like structure. In certain examples, a three-dimensional camera can be used to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 may be mounted on another surgical device instead of the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar that extends through which the surgical device 102 passes to reach the surgical site. In yet another example, the receiver 108 for the time-of-flight distance sensor system 104 may be mounted on another robotic control arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or on a table or fixture in an operating room (OR). In a particular example, the imaging device 120 includes a time-of-flight receiver 108 that determines the distance from the emitter 106 on the surgical device 102 to the surface 105 of the tissue 103 using a line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, distance d e This can be triangulated based on the known positions of the emitter 106 (on the surgical device 102) and receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. The three-dimensional position of the receiver 108 is known and / or can be aligned with respect to the robot coordinate plane during surgery.

[0044] In certain examples, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by a first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by a second robotic arm 114. In other examples, the surgical visualization system 100 may be used separately from the robotic system. In such cases, the distance sensor system 104 may be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the primary robotic system used in a surgical procedure. At least one of the robotic arms 112, 114 may be positioned and aligned in a particular coordinate system without servo motor control. For example, a closed-loop control system and / or a plurality of sensors for the robotic arm 110 can control and / or align the positions of the robotic arms 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned relative to a particular coordinate system.

[0046] Referring further to FIG. 1, d w is the camera-important structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the important structure 101, and d A is the depth of the important structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the important structure 101). In various aspects, the flight time of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-important structure distance d w . The use of spectral imaging in combination with a flight time sensor is further described herein. Further, referring now to FIG. 3, in various aspects of the present disclosure, the depth d A of the important structure 101 relative to the surface 105 of the tissue 103 can be determined by triangulation from the distance d w , and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the distance d x ) between them, and the distance d y can be determined, where the distance d y is the sum of the distances d e and d A .

[0047] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or range of waveforms) can be used to determine the camera-critical structure distance d w This can be determined, and the distance to the surface 105 of the tissue 103 can be determined using a second waveform (or range of waveforms). In such an example, the depth of the important structure 101 below the surface 105 of the tissue 103 can be determined using a different waveform.

[0048] Additionally or alternatively, in certain examples, distance d A This can be determined from ultrasound, registered magnetic resonance imaging (MRI), or computerized tomography (CT) scans. In other examples, distance d A This can be determined by spectral imaging because the detection signal received by the imaging device may change based on the type of substance. For example, fat can reduce the detection signal in a first way or by a first amount, and collagen can reduce the detection signal in a different second way or by a second amount.

[0049] Referring here to the surgical visualization system 160 in Figure 4, the surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform emitter 123 is located at a distance d from a common device such as the surgical device 162, as further described herein. t and d w It can be configured to emit a waveform for determining the distance d from the surface 105 of tissue 103 to the surface of important structure 101. A This can be determined as follows: d A =d w -d t

[0050] As disclosed herein, various information regarding visible tissue, buried vital structures, and surgical instruments can be determined by utilizing a combination method incorporating one or more time-of-flight distance sensors, spectral imaging, and / or structural light arrays, in combination with an image sensor configured to detect spectral wavelengths and structural light arrays. Furthermore, the image sensor may be configured to provide an image of the surgical site to the imaging system by receiving visible light. Logic or algorithms are used to identify the information received from the time-of-flight sensor, spectral wavelengths, structural light, and visible light, and to render three-dimensional images of surface tissue and underlying anatomical structures. In various examples, the imaging device 120 may include multiple image sensors.

[0051] Camera-to-critical structure distance d w Furthermore, it can be detected by one or more alternative methods. In one embodiment, the important structure 201, as shown in Figures 6 to 8, can be illuminated using fluorescence visualization techniques, such as fluorescent indocyanine green (ICG). The camera 220 may include two optical waveform sensors 222, 224 that simultaneously capture left and right images (Figures 7A and 7B) of the important structure 201. In such an example, the camera 220 can depict the glow of the important structure 201 beneath the surface 205 of the tissue 203, at a distance d w This can be determined by the known distance between sensor 222 and sensor 224. In certain examples, the distance can be determined more accurately by using two or more cameras or by moving the cameras between multiple positions. In certain embodiments, one camera may be controlled by a first robotic arm and a second camera by another robotic arm. In such a robotic system, one camera may be, for example, a driven camera on a driven arm. The driven arm and the camera on it can be programmed, for example, to track other cameras and maintain a specific distance and / or lens angle.

[0052] In yet another embodiment, the surgical visualization system 100 uses two separate waveform receivers (i.e., cameras / image sensors) to perform d w This can be determined. Referring to Figure 9, if an important structure 301 or its contents (e.g., blood vessels or the contents of blood vessels) can emit a signal 302 by fluoroscopy or the like, its actual position can be determined by triangulation from two separate cameras 320a and 320b at known positions.

[0053] In another embodiment, referring here to Figures 10A and 10B, the surgical visualization system uses dithering or a mobile camera 440 to measure distance d wThis can be determined. Camera 440 is robotically controlled such that its three-dimensional coordinates at different positions are known. In various examples, camera 440 can pivot on a cannula or patient interface. For example, if a critical structure 401 or its contents (e.g., a blood vessel or the contents of a blood vessel) can emit a signal, for example by fluoroscopy, its actual position can be triangulated from camera 440 rapidly moving between two or more known positions. In Figure 10A, camera 440 moves axially along axis A. More specifically, camera 440 translates a distance d1 closer to the critical structure 401 along axis A to a location indicated as position 440', for example, by moving in and out on a robotic arm. As camera 440 moves a distance d1 and the size of the field of view changes relative to the critical structure 401, the distance to the critical structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) may correspond to angles θ1 of 6.28 degrees and angles θ2 of 8.19 degrees. Additionally or alternatively, the camera 440 can rotate or sweep along an arc between different positions. Referring here to Figure 10B, the camera 440 moves axially along axis A and rotates by an angle θ3 about axis A. The pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. In Figure 10B, the camera 440 translates and rotates to position 440''. As the camera 440 moves and the edge of the image relative to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. In Figure 10B, the distance d2 may be, for example, 9.01 mm, and the angle θ3 may be, for example, 0.9 degrees.

[0054] Figure 5 depicts a surgical visualization system 500, which is similar in many respects to the surgical visualization system 100. In various examples, the surgical visualization system 500 may be a further example of the surgical visualization system 100. Like the surgical visualization system 100, the surgical visualization system 500 includes a surgical device 502 and an imaging device 520. The imaging device 520 includes a spectral light emitter 523, which is configured to emit spectral light of multiple wavelengths to acquire spectral images of hidden structures, for example. The imaging device 520 may also include a three-dimensional camera and associated electronic processing circuits in various examples. The surgical visualization system 500 is shown being used during surgery to identify and facilitate the avoidance of certain critical structures, such as the ureters 501a and blood vessels 501b within organs 503 (the uterus in this example) that are not visible from the surface.

[0055] The surgical visualization system 500 uses structural light to measure the emitter-tissue distance d from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503. e The surgical visualization system 500 is configured to determine the emitter-tissue distance d e Based on this, the distance d between the surgical instrument 502 and the surface 505 of the uterus 503 is determined by the instrument-tissue distance. t It is configured to estimate the tissue-ureteral distance d from the ureter 501a to the surface 505. A , and the camera-ureter distance d from the imaging device 520 to the ureter 501a w It is configured to determine the distance d. As described herein with respect to Figure 1, for example, the surgical visualization system 500 uses, for example, spectral imaging and time-of-flight sensors to determine the distance d w The surgical visualization system 500 can determine the tissue-ureteral distance d based on other distance and / or surface mapping logic described herein. In various examples, the surgical visualization system 500 can determine the tissue-ureteral distance d based on other distance and / or surface mapping logic described herein. A (That is, depth) can be determined (for example, by triangulation).

[0056] Referring here to Figure 11, a schematic diagram of a control system 600 for a surgical visualization system, such as surgical visualization system 100, is depicted. The control system 600 is a transformation system that integrates spectral signature tissue identification and structural optical tissue positioning to identify important structures, particularly when these structures are obscured by other tissues, such as fat, connective tissue, blood, and / or other organs. Such techniques may also be useful in detecting tissue variations, such as distinguishing tumors and / or pathological tissues from healthy tissue within an organ.

[0057] The control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are used to detect and identify anatomical structures within the surgical field. The control system 600 includes a conversion logic circuit 648 for converting tissue data into information usable by the surgeon. For example, key structures within an anatomical structure can be identified by utilizing wavelength-based variable reflectance for hidden material. Furthermore, the control system 600 combines the identified spectral signature and structural optical data in the image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical use in a system with augmented image overlays. The technology can be used both intraoperatively and preoperatively with additional visual information. In various examples, the control system 600 is configured to provide alerts to the clinician when one or more key structures are in proximity. Based on the surgical procedure and proximity to key structures, various algorithms can be employed to guide robotic and semi-automated approaches.

[0058] The projected light array is used to determine the shape and movement of tissue during surgery. Alternatively, flash lidar may be used for tissue surface mapping.

[0059] The control system 600 is configured to detect critical structures, provide image overlays of critical structures, and measure the distance to the surface of visible tissue and the distance to embedded / covered critical structures. In other examples, the control system 600 may measure the distance to the surface of visible tissue or detect critical structures and provide image overlays of critical structures.

[0060] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 may be, for example, a field programmable gate array (FPGA), or another preferred circuit configuration as described herein in relation to Figures 2A to 2C. The spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, and can utilize, for example, C / C++ code. The video input processor 606 accepts a video input terminal for control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608, which includes, for example, a hyperspectral video output terminal for interface control (metadata) data. The video output processor 608 provides the video output signal to an image overlay controller 610.

[0061] The video input processor 606 is connected to the patient-side camera 612 via a patient isolation circuit 614. As previously discussed, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include a number of transformers to isolate the patient from other circuits in the system. The camera 612 receives intraoperative images via an optical element 632 and the image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or any of the image sensor technologies discussed herein in relation to, for example, Figure 2. In one embodiment, the camera 612 outputs an image with a 14-bit / pixel signal. It will also be understood that higher or lower pixel resolutions may be utilized without departing from the scope of this disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fusion output signal is provided to the video input processor 606 and the laser pulse control circuit 622.

[0062] The laser pulse control circuit 622 controls the laser light engine 624. The laser light engine 624 emits multiple wavelengths (λ1, λ2, λ3...λ) including near-infrared (NIR). n The laser light engine 624 outputs light in a specific mode. The laser light engine 624 can operate in multiple modes. In one embodiment, the laser light engine 624 can operate in, for example, two modes. In the first mode, for example, the normal operation mode, the laser light engine 624 outputs an illumination signal. In the second mode, for example, the identification mode, the laser light engine 624 outputs RGBG light and NIR light. In various examples, the laser light engine 624 can operate in polarization mode.

[0063] The light output 626 from the laser light engine 624 illuminates a target anatomical structure within the surgical site 627 during surgery. The laser pulse control circuit 622 also controls the laser pulse controller 628 for the laser pattern projector 630, which projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, onto the surgical tissue or organs of the surgical site 627 at a predetermined wavelength (λ2). The camera 612 receives the patterned and reflected light output through the camera optical element 632. The image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and to the video input module 636 for reading the laser beam pattern 631 projected onto the target anatomical structure of the surgical site 627 by the laser pattern projector 630. The processing module 638 processes the laser beam pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendering shape of the tissue or organ of the target anatomical structure at the surgical site.

[0065] The first and second video output signals 640 and 642 contain data representing the location of critical structures on a three-dimensional surface model, which are provided to the integrated module 643. Combined with the data from the video output processor 608 of the spectral control circuit 602, the integrated module 643 can determine the distance dA (Figure 1) to the covered critical structure (e.g., via a triangulation algorithm 644), and the distance dA can be provided to the image overlay controller 610 via the video output processor 646. The aforementioned transformation logic may include the transformation logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0066] Preoperative data 650 obtained from CT or MRI scans can be used to locate or align specific three-dimensionally deformable tissues in various cases. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610, so that such information can be overlaid with images from the camera 612 and provided to the video monitor 652. The locating of preoperative data is described herein and further described in the aforementioned U.S. patent applications, for example, U.S. Patent Application No. 16 / 128,195, entitled "INTEGRATION OF IMAGING DATA," filed September 11, 2018, each of which is incorporated herein by reference in its entirety.

[0067] The video monitor 652 can output an integrated / enhanced image from the image overlay controller 610. The clinician can select and / or switch between different images on one or more monitors. On the first monitor 652a, the clinician can switch between (A) an image showing a three-dimensional rendering of visible tissue and (B) an enhanced image in which one or more hidden important structures are drawn on top of the three-dimensional rendering of visible tissue. On the second monitor 652b, the clinician can switch, for example, one or more hidden important structures and / or distance measurements to the surface of visible tissue.

[0068] The control system 600 and / or various control circuits thereof can be incorporated into various surgical visualization systems disclosed herein.

[0069] Figure 12 shows a structural (or patterned) light system 700 according to at least one aspect of the present disclosure. As described herein, structural light, for example in the form of stripes or lines, is projected from a light source and / or projector 706 onto the surface 705 of a target anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which may be similar in various respects to an imaging device 120 (Figure 1), may be configured to detect the projected light pattern on the surface 705. The projection pattern deforms upon impact with the surface 705, enabling the visual system to calculate depth and surface information of the target anatomical structure.

[0070] In certain cases, invisible (or undetectable) structured light can be used without interfering with other computer visual tasks where the projection pattern might be disruptive. For example, interference can be prevented by using infrared light or visible light at a very fast frame rate, repeating two exactly opposite patterns. Structured light is described further at en.wikipedia.org / wiki / Structured_light.

[0071] As described above, various surgical visualization systems described herein can be used to visualize various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be hidden from visualization by EMR in the visible portion of the spectrum. In one embodiment, a surgical visualization system can utilize a spectral imaging system to visualize different types of tissues based on various combinations of constituent materials. In particular, a spectral imaging system can be configured to detect the presence of various constituent materials in the tissue being visualized based on the absorption coefficient of the tissue across various EMR wavelengths. A spectral imaging system can be further configured to characterize the tissue type of the tissue being visualized based on a specific combination of constituent materials. For illustrative purposes, Graph 2300 in Figure 13A illustrates how the absorption coefficients of various biological materials change across the EMR wavelength spectrum. In Graph 2300, the vertical axis 2303 represents the absorption coefficient of the biological material (e.g., cm) -1 The graph is shown with the horizontal axis 2304 representing the EMR wavelength (e.g., in μm). Graph 2300 further shows the first line 2310 representing the absorption coefficient of water at various EMR wavelengths, the second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, the third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, the fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, the fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and the sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituents, and therefore, tissue types visualized by the surgical visualization system can be identified and distinguished according to specific combinations of constituents detected. Therefore, a spectral imaging system can be configured to emit EMR at several different wavelengths, determine the tissue components based on the absorbed EMR absorption responses detected at different wavelengths, and then characterize the tissue type based on a specific detected combination of components.

[0072] An example of the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures is shown in Figure 13B. In Figure 13B, a spectral emitter 2320 (e.g., spectral light source 150) is used by the imaging system to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissues and / or structures of the surgical site 2325 is received by an image sensor 135 (Figure 2) to visualize tissues and / or structures that may be either visible (e.g., located on the surface of the surgical site 2325) or hidden (e.g., beneath other tissues and / or structures in the surgical site 2325). In this example, the imaging system 142 (Figure 2) can visualize tumors 2332, arteries 2334, and various anomalies 2338 (i.e., tissues not identified against known or expected spectral signatures) based on spectral signatures characterized by different absorption properties (e.g., absorption coefficients) of each constituent material of different tissue / structure types. The visualized tissues and structures can be displayed on display screens associated with or connected to the imaging system 142, such as the imaging system display 146 (Figure 2), primary display 2119 (Figure 18), non-sterile display 2109 (Figure 18), hub display 2215 (Figure 19), and device / instrument display 2237 (Figure 19).

[0073] Furthermore, the imaging system 142 can be configured to adjust or update the displayed surgical site visualization according to the identified tissue and / or structural type. For example, the imaging system 142 can display a margin 2330a associated with a tumor 2332 visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue to be excised to ensure complete removal of the tumor 2332. The control system 133 (Figure 2) can be configured to control or update the dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 identifies multiple anomalies 2338 within the FOV. Therefore, the control system 133 can adjust the displayed margin 2330a to a first updated margin 2330b having sufficient dimensions to encompass the anomalies 2338. Furthermore, the imaging system 142 also identifies artery 2334 that partially overlaps with the initially displayed margin 2330a (as indicated by the highlighted region 2336 of artery 2334). Thus, the control system 133 can adjust the displayed margin 2330a to a second updated margin 2330c that has sufficient dimensions to encompass the relevant portion of artery 2334.

[0074] Tissues and / or structures can also be imaged or characterized across the EMR wavelength spectrum according to their reflectivity, in addition to or instead of their absorption characteristics as described above with respect to Figures 13A and 13B. For example, Figures 13C–13E show various graphs of the reflectivity of different types of tissues or structures across different EMR wavelengths. Figure 13C is an exemplary ureteral signature versus camouflage graph 1050. Figure 13D is an exemplary arterial signature versus camouflage graph 1052. Figure 13E is an exemplary nerve signature versus camouflage graph 1054. The plots in Figures 13C–13E represent the reflectivity as a function of wavelength (nm) for specific structures (ureters, arteries, and nerves) compared to the corresponding reflectivity of fat, lung tissue, and blood at the corresponding wavelengths. It should be understood that these graphs are for illustrative purposes only, and other tissues and / or structures may have corresponding detectable reflectivity signatures that would allow for the identification and visualization of the tissues and / or structures.

[0075] In various applications, the selective wavelength for spectral imaging can be identified and utilized based on anticipated important structures and / or obstructions at the surgical site (i.e., "selective spectral" imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized, thereby enabling the acquisition of information in real-time or near real-time for intraoperative use. In various applications, the wavelength can be selected by the clinician or by a control circuit based on clinician input. In specific applications, the wavelength can be selected based on machine learning and / or big data accessible to the control circuit, for example, via the cloud.

[0076] The application of spectral imaging to the tissue described above can be used intraoperatively to measure the distance between a waveform emitter and critical structures obscured by the tissue. In one aspect of this disclosure, with reference to Figures 14 and 15, a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown. In certain examples, the time-of-flight sensor system 1104 can be incorporated into a surgical visualization system 100 (Figure 1). The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted wave 1124 extends from the emitter 1106 to the critical structure 1101, and the received wave 1125 is reflected back from the critical structure 1101 by the receiver 1108. The surgical device 1102 is positioned through a trocar 1110 extending within the patient's cavity 1107.

[0077] Waveforms 1124 and 1125 are configured to penetrate the obscuring tissue 1103. For example, the wavelengths of waveforms 1124 and 1125 may be wavelengths of the NIR spectrum or SWIR spectrum. In one embodiment, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal can be emitted from the emitter 1106 and penetrate the tissue 1103 obscuring the critical structure 1101. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various examples, waveforms 1124 and 1125 may be selected to target the critical structure 1101 in the tissue 1103 based on the spectral signature of the critical structure 1101, as further described herein. In various examples, the emitter 1106 is configured to provide on and off binary signals, for example, as shown in Figure 15, which can be measured by the receiver 1108.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (Figure 14). The time-of-flight timing of the emitter 1106 and receiver 1108 in Figure 14 is shown in Figure 15. The delay is a function of the distance d, and the distance d is given by the following equation:

[0079]

number

[0080] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to distance d in Figure 14. In various examples, an additional emitter / receiver and / or pulse signal from emitter 1106 can be configured to emit an opaque signal. The opaque structure can be configured to determine the distance from the emitter to the surface 1105 of the covering structure 1103. In various examples, the depth of critical structure 1101 can be determined by the following formula. d A =d w -d t During the ceremony, d A =This is the depth of important structure 1101, d w = This is the distance from emitter 1106 to critical structure 1101 (d in Figure 14), d t = This is the distance from the emitter 1106 (on the distal end of the surgical instrument 1102) to the surface 1105 of the covering tissue 1103.

[0081] In one aspect of this disclosure, with reference to Figure 16, a time-of-flight sensor system 1204 is shown that utilizes waves 1224a, 1224b, 1224c, 1225a, 1225b, and 1225c. In certain examples, the time-of-flight sensor system 1204 can be incorporated into a surgical visualization system 100 (Figure 1). The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is positioned on a first surgical device 1202a, and the waveform receiver 1208 is positioned on a second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their corresponding trocars 1210a and 1210b, respectively, which extend within a patient cavity 1207. The emitted waves 1224a, 1224b, and 1224c extend from the emitter 1206 toward the surgical site, and the received waves 1225a, 1225b, and 1225c are reflected back to the receiver 1208 from various structures and / or surfaces at the surgical site.

[0082] Different emitted waves 1224a, 1224b, and 1224c are configured to target different types of material at the surgical site. For example, wave 1224a targets the covering tissue 1203, wave 1224b targets a first important structure 1201a (e.g., blood vessels), and wave 1224c targets a second important structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, and 1224c may be wavelengths of visible light, NIR, or SWIR spectra. For example, visible light can be reflected by the surface 1205 of the tissue 1203, and NIR and / or SWIR waveforms can be configured to transmit through the surface 1205 of the tissue 1203. In various embodiments, spectral signals (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signals can be emitted from the emitter 1206 as described herein. In various examples, waves 1224b and 1224c may be selected to target key structures 1201a and 1201b within tissue 1203 based on the spectral signatures of key structures 1201a and 1201b, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications incorporated herein by reference.

[0083] The emitted waves 1224a, 1224b, and 1224c can be reflected from the target material (i.e., surface 1205, first key structure 1201a, and second structure 1201b, respectively). The received waveforms 1225a, 1225b, and 1225c are reflected at the distance d shown in Figure 16. 1a d 2a d 3a d 1b d 2b d 2c This may cause delays.

[0084] In a time-of-flight sensor system 1204 in which the emitter 1206 and receiver 1208 can be independently positioned (for example, on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), the emitter 1206 and receiver 1208 can be positioned at various distances d from a known position. 1a d2a d 3a d 1b d 2b d 2c It is possible to calculate the distance d. For example, when surgical devices 1202a and 1202b are robotically controlled, their positions may be known. Knowledge of the positions of emitters 1206 and receivers 1208, as well as the time of the photon stream targeting a particular tissue and the information of that particular response received by receiver 1208, allows us to calculate the distance d. 1a d 2a d 3a d 1b d 2b d 2c This makes it possible to determine the distance. In one embodiment, the distance to the obscured critical structures 1201a and 1201b can be determined by triangulation using the transmitted wavelength. Since the speed of light is constant for all wavelengths of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0085] Referring further to Figure 16, in various examples, in the image provided to the clinician, the receiver 1208 can be rotated such that the center of mass of the target structure in the resulting image remains constant, i.e., it lies in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such orientation can quickly communicate one or more associated distances and / or viewpoints to the important structure. For example, as shown in Figure 16, the surgical site is displayed from a viewpoint where the important structure 1201a is perpendicular to the field of view plane (i.e., the blood vessels face in opposite directions on the paper). In various examples, such orientation may be the default setting. However, the image may be rotated or otherwise adjusted by the clinician. In certain examples, the clinician can switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 may be mounted on a trocar or cannula, such as a trocar 1210b, through which the surgical instrument 1202b is positioned. In other examples, the receiver 1208 may be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 may be mounted on a movable arm separate from the robot controlling the surgical instrument 1202a, or on a table in the operating room (OR) that can be aligned to the robot coordinate plane during surgery. In such examples, the positions of the emitter 1206 and the receiver 1208 can be aligned to the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] A combination of a time-of-flight sensor system called TOF-NIRS and near-infrared spectroscopy (NIRS), which enables the measurement of the time-resolved properties of NIR light with nanosecond resolution, can be found in the Journal of the American Society for Horticultural Science, May 2013, vol. 138, no. 3, pp. 225-228, entitled "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT," which is incorporated herein by reference in its entirety and is accessible at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, time-of-flight spectral waveforms are configured to determine the depth of critical structures and / or the proximity of surgical instruments to critical structures. Furthermore, various surgical visualization systems disclosed herein include surface mapping logic configured to produce a three-dimensional rendering of the surface of visible tissue. In such examples, even if the visible tissue obscures the critical structure, the clinician can recognize the proximity (or lack thereof) of the surgical instrument to the critical structure. In one example, a topography of the surgical site is provided on a monitor by surface mapping logic. If the critical structure is close to the surface of the tissue, spectral imaging can communicate the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 or 10 mm of the surface. In other examples, spectral imaging can detect structures 10 or 20 mm below the surface of the tissue. Based on known limits of the spectral imaging system, the system is configured to indicate that a critical structure is out of range if it is simply not detected by the spectral imaging system. Thus, the clinician can continue to move the surgical instrument and / or continue to manipulate the tissue. When a critical structure moves within the range of a spectral imaging system, the system can identify the structure and therefore communicate that the structure is within range. In such cases, an alert can be provided when the structure is first identified and / or when it moves further into a predetermined proximity zone. In such cases, even if the critical structure is not identified by a spectral imaging system with known boundaries / ranges, the clinician can be provided with proximity information (i.e., not in proximity).

[0089] Various surgical visualization systems disclosed herein may be configured to identify the presence and / or proximity of critical structures during surgery and to alert clinicians before damage to critical structures occurs due to inadvertent incision and / or cutting. In various embodiments, surgical visualization systems are configured to identify one or more of the following important structures, for example, ureters, intestines, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve (RLN), process facial nerve, vagus nerve, and their bifurcations), blood vessels (including arteries and veins of the lungs and lobes, inferior mesenteric artery (IMA) and its bifurcations, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its bifurcations (including the middle colic artery, right colic artery, and ileocolic artery), hepatic arteries and their bifurcations, portal veins and their bifurcations, splenic arteries / veins and their bifurcations, external and internal iliac vessels (lower abdomen), short gastric arteries, uterine arteries, median sacral vessels, and lymph nodes. Furthermore, the surgical visualization system is configured to indicate the proximity of surgical instruments to critical structures and / or to warn the clinician when the surgical instruments approach critical structures.

[0090] Various aspects of this disclosure provide identification of critical structures during surgery (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking that enable visualization of critical structures below the surface of tissue, such as 1.0 to 1.5 cm below the tissue surface. In other examples, surgical visualization systems can identify structures less than 1.0 cm or more than 1.5 cm below the tissue surface. For example, even a surgical visualization system that can identify only structures within 0.2 mm of the surface may be useful, for example, when structures are not normally visible due to depth. In various aspects, surgical visualization systems can extend the clinician's field of view by, for example, virtually displaying critical structures as an overlay of a visible white light image on the surface of visible tissue. Surgical visualization systems can provide real-time three-dimensional spatial tracking of the distal tip of a surgical instrument and can provide proximity alerts when the distal tip of a surgical instrument moves within a range of a critical structure, for example, within 1.0 cm of the critical structure.

[0091] Various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. An incision may be "too close" to a critical structure based on temperature (i.e., excessively hot near the critical structure, potentially damaging / heating / melting the critical structure) and / or tension (i.e., excessively high tension near the critical structure, potentially damaging / tearing / pulling the critical structure). Such surgical visualization systems can facilitate perivascular incisions, for example, when dissecting perivascular tissue before ligation. In various examples, a thermal imaging camera can be used to read the heat of the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the tool temperature exceeds a predetermined threshold (e.g., 120°F), an alert may be provided to the clinician at a first distance (e.g., 10 mm), and if the tool temperature is below a predetermined threshold, an alert may be provided to the clinician at a second distance (e.g., 5 mm). Default thresholds and / or warning distances may be default settings and / or programmable by the clinician. Additionally or alternatively, proximity alerts may be linked to thermal measurements performed by the tool itself, such as thermocouples measuring heat in the distal jaws of unipolar or bipolar cutting instruments or vascular sealers.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity to critical structures to enable clinicians to confidently and rapidly but safely perform incisions based on caution criteria and / or device safety data. These systems can function intraoperatively in real time during surgical procedures with minimal ionizing radiation risk to the patient or clinician, and in various cases, there is no ionizing radiation risk to the patient or clinician. Conversely, in fluoroscopy, patients and clinicians may be exposed to ionizing radiation, for example, via the X-ray beam used to visualize anatomical structures in real time.

[0093] Various surgical visualization systems disclosed herein may be configured to detect and identify one or more desired types of critical structures in the forward path of a surgical device, for example, when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system may be configured to detect and identify one or more types of critical structures in the surrounding area and / or multiple planes / ranges of the surgical device, for example.

[0094] The various surgical visualization systems disclosed herein are easily operable and / or interpretable. Furthermore, the various surgical visualization systems may incorporate “override” features that allow clinicians to override default settings and / or operations. For example, when the risk to a critical structure is lower than the risk of avoiding that area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue may be greater than the risk of damaging the critical structure), the clinician may selectively turn off alerts from the surgical visualization system and / or approach the critical structure more closely than the surgical visualization system presents.

[0095] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on the workflow. In other words, the implementation of a surgical visualization system does not require a change in the way the surgical procedure is performed. Furthermore, surgical visualization systems may be more economical compared to the cost of accidental amputation. Data show that this can be gradually recouped by reducing accidental damage to critical structures.

[0096] The various surgical visualization systems disclosed herein can operate in real time or near real time and well in advance to enable clinicians to anticipate critical structures. For example, a surgical visualization system can provide sufficient time for "slowing down, evaluating, and avoiding" to maximize the efficiency of surgical procedures.

[0097] Various surgical visualization systems disclosed herein do not require contrast agents or dyes to be injected into the tissue. For example, spectral imaging is configured to visualize structures hidden during surgery without the use of contrast agents or dyes. In other examples, contrast agents may be easier to inject into the appropriate layers of tissue than other visualization systems. The time between contrast agent injection and visualization of important structures may be, for example, less than two hours.

[0098] The various surgical visualization systems disclosed herein can be linked with clinical data and / or instrument data. For example, the data can provide boundaries regarding the distance from tissue that the surgeon does not want to damage to an energy-activated surgical instrument (or other potentially damaging instrument). Any data modules that interface with the surgical visualization systems disclosed herein may be provided in conjunction with a robot or separately for use with a standalone surgical instrument, for example, in invasive or laparoscopic procedures. The surgical visualization systems can be adapted with robotic surgical systems in various examples. For example, the visualization images / information may be displayed within the robot console.

[0099] In various cases, clinicians may not know the location of critical structures relative to surgical tools. For example, if critical structures are embedded in tissue, it may be impossible for clinicians to locate them. In certain cases, clinicians may want to keep surgical instruments outside the range surrounding critical structures and / or away from visible tissue covering hidden critical structures. If the location of hidden critical structures is unknown, clinicians risk getting too close to them, potentially resulting in inadvertent trauma and / or incision of critical structures, and / or excessive energy, heat, and / or tension near them. Alternatively, clinicians may try to avoid critical structures by remaining far away from suspected critical structures, risking affecting tissue in undesirable locations.

[0100] A surgical visualization system is provided that presents tracking of surgical instruments to one or more critical structures. For example, the surgical visualization system can track the proximity of surgical instruments to critical structures. Such tracking can be performed in real time and / or near real time during surgery. In various examples, the tracking data may be provided to the clinician via the display screen (e.g., monitor) of the imaging system.

[0101] In one aspect of this disclosure, a surgical visualization system comprises a surgical device having an emitter configured to emit a structural light pattern onto a visible surface; an imaging system having a camera configured to detect a buried structure and the structural light pattern on a visible surface; and a control circuit communicating with the camera and the imaging system, wherein the control circuit is configured to determine the distance from the surgical device to the buried structure and to provide the imaging system with a signal indicating the distance. For example, the distance can be determined by calculating the distance from the camera to a critical structure that glows by fluorescence fluoroscopy and based on a three-dimensional image of the glowing structure provided by images from multiple lenses of the camera (e.g., a left lens and a right lens). The distance from the surgical device to the critical structure can be determined, for example, by triangulation based on known positions of the surgical device and the camera. Alternative means for determining the distance to a buried critical structure are further described herein. For example, an NIR time-of-flight distance sensor can be used. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue overlapping / covering the buried critical structure. For example, a surgical visualization system can identify hidden critical structures and enhance images of these structures by drawing outlines of them on the visible tissue, such as lines on the surface of the visible tissue. The surgical visualization system can further determine the distance to these enhanced lines on the visible tissue.

[0102] As provided by the various surgical visualization systems disclosed herein, clinicians can make more informed decisions regarding the placement of surgical instruments relative to hidden critical structures by providing them with up-to-date information on the proximity of surgical instruments to hidden critical structures and / or visible structures. For example, clinicians can view the distance between surgical instruments and critical structures in real time / intraoperatively, and in certain cases, the imaging system can provide alerts and / or warnings when the surgical instrument moves within a predetermined proximity and / or zone of a critical structure. In certain cases, alerts and / or warnings may be provided when the trajectory of the surgical instrument indicates a potential collision with a “no-fly” zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more of the critical structure). In such cases, clinicians can maintain the operation through the surgical procedure without needing to monitor the location of suspected critical structures and the proximity of the surgical instrument to them. As a result, certain surgical procedures can be performed more quickly, for example, with fewer interruptions and / or with improved accuracy and / or certainty. In one embodiment, a surgical visualization system can be used to detect tissue diversity, such as tissue diversity within an organ, in order to distinguish tumor / cancerous / pathogenic tissue from healthy tissue. Such a surgical visualization system can maximize the removal of pathological tissue while minimizing the removal of healthy tissue.

[0103] Surgical Hub System The various visualization or imaging systems described herein are illustrated in relation to Figures 17-19 and can be incorporated into surgical hub systems as described in more detail below.

[0104] Referring to Figure 17, the computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 which may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 which may include the remote server 2113. In one embodiment, as shown in Figure 17, the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112, which are configured to communicate with each other and / or with the hub 2106. In some embodiments, the surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers of 1 or more.

[0105] Figure 18 depicts an embodiment of a surgical system 2102 used to perform surgical procedures on a patient lying on an operating table 2114 in a surgical operating room 2116. A robotic system 2110 is used in the surgical procedure as a component of the surgical system 2102. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (surgical robot), and a surgical robot hub 2122. The patient-side cart 2120 allows the surgeon to manipulate at least one detachably connected surgical tool 2117 through a minimally invasive incision in the patient's body while viewing the surgical site via the surgeon's console 2118. Images of the surgical site can be acquired by a medical imaging device 2124, which can be operated by the patient-side cart 2120 to orient the imaging device 2124. The robotic hub 2122 can be used to process images of the surgical site, and the processed images can then be displayed to the surgeon via the surgeon's console 2118.

[0106] Other types of robotic systems can be readily adapted for use with surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0107] Various examples of cloud-based analysis performed by Cloud 2104 and suitable for use with this disclosure are described in various U.S. patent applications, which are incorporated herein by reference.

[0108] In various embodiments, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0109] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes called the optical spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0111] The invisible spectrum (i.e., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0112] In various embodiments, the imaging device 2124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, gastroscopy (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0113] In one embodiment, the imaging device employs multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments capable of sensing specific wavelengths of light, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging can enable the extraction of additional information that the human eye cannot capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated herein by reference. Multispectral monitoring can be a useful tool for performing one or more of the above-described tests on the tissue being treated, and for repositioning the surgical field after the completion of a surgical task.

[0114] It is self-evident that strict sterilization of the operating room and surgical equipment is necessary in any surgical procedure. The strict hygiene and sterilization conditions required in the "surgical theater," i.e., the operating room or treatment room, require the highest level of sterilization for all medical devices and equipment. Part of the above sterilization process requires sterilizing everything that comes into contact with the patient or enters the sterile field, such as the imaging device 2124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or it may be considered the area immediately surrounding the patient prepared for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and restraints within that area.

[0115] In various embodiments, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically positioned relative to a sterile field, as shown in Figure 18. In one embodiment, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 2108 are described in various U.S. patent applications, which are incorporated herein by reference.

[0116] As shown in Figure 18, the primary display 2119 is positioned within the sterile field on the operating table 2114 so that it is visible to the operator. In addition, the visualization tower 21121 is positioned outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109, which face in opposite directions to each other. The visualization system 2108, guided by the hub 2106, is configured to utilize displays 2107, 2109, and 2119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 2106 may cause the visualization system 2108 to display snapshots of the surgical site on the non-sterile displays 2107 or 2109 as recorded by the imaging device 2124, while maintaining a live image of the surgical site on the primary display 2119. The snapshots on the non-sterile displays 2107 or 2109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0117] In one embodiment, the hub 2106 is also configured to route diagnostic input or feedback entered by a non-sterile operator in the visualization tower 21121 to a primary display 2119 in the sterile area, which can be viewed by a sterile operator on the operating table. In one embodiment, the input may take the form of modifications to snapshots displayed on the non-sterile displays 2107 or 2109, which can be routed to the primary display 2119 by the hub 2106.

[0118] Referring to Figure 18, the surgical instrument 2112 is used in a surgical procedure as part of the surgical system 2102. The hub 2106 is also configured to coordinate the flow of information to the display of the surgical instrument 2112, as described in various U.S. patent applications incorporated herein by reference. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 21121 may be routed by the hub 2106 to the surgical instrument display 2115 in the sterile field, where the operator of the surgical instrument 2112 can view this diagnostic input or feedback. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. patent applications incorporated herein by reference.

[0119] Figure 19 shows a computer-implemented interactive surgical system 2200. The computer-implemented interactive surgical system 2200 is similar in many respects to the computer-implemented interactive surgical system 2100. The surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204, which may include a remote server 2213. In one embodiment, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for connecting the surgical hub 2236 to the cloud 2204 and / or the remote server 2213 in a communicative manner. As shown in the example in Figure 19, the surgical hub 2236 is connected to an imaging module 2238 connected to an endoscope 2239, a generator module 2240 connected to an energy device 2421, a fume exhaust module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235 optionally connected to a display 2237, and a non-contact sensor module 2242. The operating room equipment is connected to cloud computing resources and data storage via the surgical hub 2236. The robot hub 2222 can also be connected to the surgical hub 2236 and cloud computing resources. In particular, the device / instrument 2235 and the visualization system 2209 can be connected to the surgical hub 2236 via wired or wireless communication standards or protocols as described herein. The surgical hub 2236 can be connected to the hub display 2215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, instrument / device display, and / or other visualization system 208. The hub display may also display data received from devices connected to the modular control tower, along with the images and overlay images.

[0120] Situational awareness Various visualization systems or embodiments of visualization systems described herein may be used as part of a situational awareness system that can be embodied or performed by the surgical hubs 2106, 2236 (Figures 17-19). In particular, by characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and movement), tissues, structures, users, and other things located in the surgical field or operating room, contextual data that can be used by the situational awareness system can be provided to estimate the type or steps of the surgical procedure to be performed, the type of tissue and / or structure being manipulated by the surgeon, etc. The situational awareness system can then use this contextual data to provide warnings to the user, provide alerts to the user, suggest subsequent steps or actions for the user to take, prepare surgical devices in anticipation of use (e.g., activating an electrosurgical generator in anticipation of an electrosurgical instrument to be used in a subsequent step of the surgical procedure), intelligently control surgical instruments (e.g., customizing the operating parameters of surgical instruments based on each patient's specific health profile), etc.

[0121] An "intelligent" device that includes a control algorithm that responds to sensed data may be an improvement over a "data-dumb" device that operates without considering sensed data, however some sensed data, when considered in isolation, may be incomplete or inconclusive without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the procedure context (e.g., knowing the type of tissue being operated on or the type of procedure being performed), a control algorithm may improperly or suboptimally control a modular device given sensed data without specific context. A modular device may include any surgical device controllable by a situation-aware system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a fume extractor). For example, the optimal way in which a control algorithm controls a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., resistance to tearing) and therefore respond differently to actions taken by the surgical instrument. Therefore, even when the same measurement is perceived for a particular parameter, it may be desirable for surgical instruments to behave differently. As one specific example, the optimal way for a surgical stapling and cutting instrument to control itself in response to sensing an unexpectedly high force to close its end effector depends on whether the type of tissue is susceptible to or resistant to tearing. For tear-sensitive tissues, such as lung tissue, the instrument's control algorithm optimally ramps down the motor in response to an unexpectedly high force to close in order to avoid tearing the tissue. For tear-resistant tissues, such as stomach tissue, the instrument's control algorithm optimally ramps up the motor in response to an unexpectedly high force to close in order to ensure that the end effector is properly clamped to the tissue.Without knowing whether lung tissue or stomach tissue is being clamped, the control algorithm may make a suboptimal decision.

[0122] One solution utilizes a surgical hub, which includes a system configured to derive information about a surgical procedure being performed based on data received from various data sources, and then appropriately control paired modular devices. In other words, the surgical hub is configured to infer information about a surgical procedure from received data, and then control the modular devices paired with the surgical hub based on the inferred context of the surgical procedure. Figure 20 shows a diagram of a context-aware surgical system 2400 according to at least one aspect of the present disclosure. In some examples, the data source 2426 includes, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database containing patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor).

[0123] The surgical hub 2404 may be similar in many ways to the hub 106, but may be configured to derive contextual information about a surgical procedure from the data based on a particular combination of received data or a particular order in which the data is received from the data source 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure to be performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This ability of the surgical hub 2404 to derive or infer information related to a surgical procedure from received data may be referred to as “situational awareness.” In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to a surgical procedure from received data.

[0124] The situational awareness system of the surgical hub 2404 may be configured to derive contextual information from data received from the data source 2426 in various different ways. In one example, the situational awareness system includes a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular device 2402) with corresponding contextual information about a surgical procedure. In other words, the machine learning system may be trained to accurately derive contextual information about a surgical procedure from the provided inputs. In another example, the situational awareness system may include a lookup table that stores pre-characterized contextual information about a surgical procedure, associating it with one or more inputs (or ranges of inputs) that correspond to the contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situational awareness system to control the modular device 2402. In one example, contextual information received by the situation awareness system of the surgical hub 2404 is associated with a specific control adjustment or a set of control adjustments for one or more modular devices 2402. In another example, the situation awareness system includes a further machine learning system, a lookup table, or other such system that, given contextual information as input, generates or retrieves one or more control adjustments for one or more modular devices 2402.

[0125] The surgical hub 2404, which incorporates a situational awareness system, offers several benefits to the surgical system 2400. One benefit is improved interpretation of the sensed and collected data, resulting in improved processing accuracy and / or data use during the course of the surgical procedure. Returning to the previous embodiment, the situational awareness surgical hub 2404 may determine what type of tissue is being operated on, and therefore, if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 2404 may precisely ramp up or ramp down the motor of the surgical instrument to match the type of tissue.

[0126] In another embodiment, the type of tissue being operated on may influence the adjustments made to the compression rate and load threshold of the surgical stapler and cutting instrument for specific interstitial space measurement. The situational awareness surgical hub 2404 can estimate whether the surgical procedure being performed is a thoracic or abdominal procedure, thereby enabling the surgical hub 2404 to determine whether the tissue clamped by the end effector of the surgical stapler and cutting instrument is lung tissue (in the case of thoracic surgery) or gastric tissue (in the case of abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical stapler and cutting instrument to match the type of tissue.

[0127] In yet another embodiment, the type of body cavity being operated on during an aerated procedure may affect the function of the smoke exhauster. The situation-aware surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing aerated air) and determine the type of procedure. Since the type of procedure is generally performed in a specific body cavity, the surgical hub 2404 can appropriately control the motor rate of the smoke exhauster to match the body cavity being operated on. In this way, the situation-aware surgical hub 2404 can provide a consistent amount of smoke exhaust for both thoracic and abdominal procedures.

[0128] In yet another embodiment, the type of procedure being performed may affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situation-aware surgical hub 2404 can determine whether the surgical procedure is an arthroscopy. The surgical hub 2404 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situation-aware surgical hub 2404 can determine what type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure. Furthermore, the situational awareness surgical hub 2404 may be configured to adjust the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument not merely for each procedure, but throughout the entire surgical procedure. The situational awareness surgical hub 2404 can determine which step of the surgical procedure is being performed or will be performed thereafter, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy levels to values ​​appropriate for the expected tissue type according to the surgical procedure step.

[0129] In yet another embodiment, data may be drawn from an additional data source 2426 to improve the conclusions that the surgical hub 2404 draws from one data source 2426. The contextually aware surgical hub 2404 can augment the data received from the modular device 2402 with contextual information constructed from other data sources 2426 regarding the surgical procedure. For example, the contextually aware surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may not be conclusive. Therefore, in one example, the surgical hub 2404 may be further configured to determine the integrity of a staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicably connected to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 2404). In other words, the contextual awareness system of the surgical hub 2404 can provide additional context when analyzing visualization data by considering physiological measurement data. This additional context can be useful when the visualization data itself may not be definitive or may be incomplete.

[0130] Another benefit is that the paired Joule device 2402 can be actively and automatically controlled according to specific steps of the surgical procedure being performed, reducing the number of times medical personnel are required to interact with or control the surgical system 2400 during the course of the surgical procedure. For example, the situation-aware surgical hub 2404 can proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that a subsequent step of the procedure requires the use of the instrument. By proactively activating the energy source, the instrument can be ready for use as soon as the preceding step of the procedure is completed.

[0131] In another embodiment, the situational awareness surgical hub 2404 can determine whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display, according to the characteristics of the surgical site that the surgeon is expected to need to visualize. The surgical hub 2404 can then proactively change the displayed view (supplied, for example, by a medical imaging device for the visualization system 108) so that the display automatically adapts throughout the surgical procedure.

[0132] In yet another embodiment, the situation-aware surgical hub 2404 can determine which steps of a surgical procedure are being performed or will be performed, and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 2404 may be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to request specific information.

[0133] Another benefit is the ability to check for errors during or in the course of a surgical procedure. For example, the situation-aware surgical hub 2404 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 may be configured to determine the type of surgical procedure being performed, retrieve the corresponding checklist, product location, or setup needs (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 may be configured to compare, for example, a list of items for the procedure scanned by a suitable scanner, and / or a list of devices paired with the surgical hub 2404, to a recommended or expected catalog of items and / or devices for a given surgical procedure. If any discontinuities exist between the lists, the surgical hub 2404 may be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical items are missing. For example, the surgical hub 2404 may be configured to determine the relative distance or position of the modular device 2402 and the patient monitoring device 2424, for instance, by proximity sensors. The surgical hub 2404 may compare the relative positions of the devices to a recommended or expected layout for a particular surgical procedure. If any discontinuity exists between the layouts, the surgical hub 2404 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout.

[0134] In another embodiment, the situation-aware surgical hub 2404 can determine whether a surgeon (or other medical professional) is making an error or otherwise deviating from the expected course of action during the surgical procedure. For example, the surgical hub 2404 may be configured to determine the type of surgical procedure being performed, read a corresponding list of equipment usage steps or sequences (e.g., from memory), and then compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 may be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0135] Overall, the context-aware system for the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) to the specific context of each surgical procedure (such as adjusting for different tissue types) and by verifying their operation during the procedure. The context-aware system also improves surgeon efficiency when performing surgical procedures by automatically suggesting the next steps, providing data, and adjusting the in-situ displays and other modular devices 2402 according to the specific context of the procedure.

[0136] Referring to Figure 21, a timeline 2500 is depicted illustrating the situational awareness of a hub, such as a surgical hub 106 or 206 (Figures 1-11). The timeline 2500 consists of an exemplary surgical procedure and the contextual information that surgical hubs 106 and 206 can derive from data received from data sources at each step of the surgical procedure. The timeline 2500 depicts typical steps that would be performed by nurses, surgeons, and other healthcare personnel during a lung segmentectomy procedure, starting with setting up the operating room and ending with transferring the patient to the postoperative recovery room.

[0137] The situation-aware surgical hubs 106, 206 receive data from data sources, including data generated each time a healthcare professional uses a modular device paired with the surgical hubs 106, 206 throughout the course of a surgical procedure. The surgical hubs 106, 206 receive this data from the paired modular devices and other data sources. As new data is received, such as which step of the procedure is being performed at any given time, it can continuously derive estimations (i.e., contextual information) about the procedure in progress. The situation-aware system of the surgical hubs 106, 206 can, for example, record data about the procedure to generate a report, verify the steps being taken by the healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular devices based on context (e.g., activating a monitor, adjusting the field of view (FOV) of a medical imaging device, or changing the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and perform any other such actions described above.

[0138] As the first step in this exemplary procedure, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hubs determine that the procedure to be performed is a thoracic procedure.

[0139] In the second step 2504, the staff scans the medical supplies arriving for the procedure. The surgical hubs 106 and 206 cross-reference the scanned supplies with a list of supplies used in various types of procedures to confirm that the combination of supplies corresponds to a thoracic procedure. Furthermore, the surgical hubs 106 and 206 can also determine that the procedure is not a wedge procedure (because the arriving supplies either do not contain specific supplies required for a thoracic wedge procedure, or are otherwise not corresponding to a thoracic wedge procedure).

[0140] In the third step 2506, the medical personnel scan the patient band with a scanner that is communicatively connected to the surgical hubs 106, 206. The surgical hubs 106, 206 can then verify the patient's identification information based on the scanned data.

[0141] In the fourth step 2508, the medical staff turns on the auxiliary devices. The auxiliary devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, the auxiliary devices include a fume extractor, an inhaler, and a medical imaging device. Once the auxiliary devices are activated, the modular devices can automatically pair with surgical hubs 106, 206 located within a specific vicinity of the modular devices as part of their initialization process. Subsequently, the surgical hubs 106, 206 can derive contextual information about the surgical procedure by detecting the type of modular device paired during this preoperative or initialization phase. In this particular embodiment, the surgical hubs 106, 206 determine that the surgical procedure is a VATS surgery based on this specific combination of paired modular devices. Based on the data from the patient's EMR, a list of medical supplies used in the surgery, and the combination of types of modular devices connected to the hubs, the surgical hubs 106, 206 can roughly estimate the specific procedure performed by the surgical team. When the surgical hubs 106 and 206 recognize what particular procedure is being performed, they then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources (e.g., modular devices and patient monitoring devices) to estimate which steps of the surgical procedure the surgical team is performing.

[0142] In the fifth step 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hubs 106 and 206. Once the surgical hubs 106 and 206 begin receiving data from the patient monitoring devices, they confirm that the patient is in the operating room.

[0143] In the sixth step 2512, medical personnel administer anesthesia to the patient. The surgical hubs 106, 206 can estimate that the patient is under anesthesia based on data from modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Upon completion of the sixth step 2512, the preoperative portion of the lung segmentectomy is completed, and the surgical portion begins.

[0144] In the seventh step 2514, the lung of the patient undergoing surgery is collapsed (while breathing is switched to the contralateral lung). The surgical hubs 106, 206 can, for example, infer from the ventilator data that the patient's lung has collapsed. Since the surgical hubs 106, 206 can compare the detection of lung collapse with the expected steps of the procedure (which can be accessed or read in advance), they can infer that the surgical portion of the procedure has already commenced and determine that collapsing the lung is the first surgical step in this particular procedure.

[0145] In the eighth step 2516, a medical imaging device (e.g., a scope) is inserted, and video from the medical imaging device is started. The surgical hubs 106, 206 receive medical imaging device data (i.e., video or image data) through their connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hubs 106, 206 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hubs 106, 206 can determine that the particular procedure being performed is a segmentectomy rather than a lobectomy (note that wedge procedures have already been determined not to be possible by the surgical hubs 106, 206 based on the data received in the second step 2504 of the procedure). Data from the medical imaging device 124 (Figure 2) may be used to determine contextual information about the type of procedure being performed in many different ways, for example, by determining the angle at which the medical imaging device is oriented relative to a visualization of the patient's anatomical structure, by monitoring the number of medical imaging devices being used (i.e., activated and paired with surgical hubs 106, 206), and by monitoring the type of visualization device being used. For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while one technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. For example, using pattern recognition or machine learning techniques, a context recognition system may be trained to recognize the position of the medical imaging device based on a visualization of the patient's anatomical structure. In another embodiment, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. In yet another example, one technique for performing VATS segmentectomy uses an infrared light source (which can be communicably connected to a surgical hub as part of a visualization system) to visualize the segmental fissure, but the infrared light source is not used in VATS lobectomy.By tracking any or all of this data from the medical imaging device, the surgical hubs 106, 206 can determine the specific type of surgical procedure being performed and / or the techniques being used in that specific type of surgical procedure.

[0146] In step 9, 2518, the surgical team initiates the incision step of the procedure. The surgical hubs 106, 206 receive data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and mobilizing the patient's lung. The surgical hubs 106, 206 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the previously considered steps of the procedure have been completed) corresponds to the incision step. In certain examples, the energy instrument may be an energy tool mounted on the robotic arm of a robotic surgical system.

[0147] In step 10, 2520, the surgical team proceeds to the ligation step of the procedure. The surgical hubs 106, 206 receive data from the surgical stapling and cutting instruments indicating that the instruments are being fired, so it can be inferred that the surgeon is currently ligating arteries and veins. As in the previous step, the surgical hubs 106, 206 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps in the read-out process. In a particular example, the surgical instruments may be surgical tools mounted on the robotic arm of a robotic surgical system.

[0148] In step 11, 2522, the segmental resection portion of the procedure is performed. The surgical hubs 106, 206 can infer that the surgeon has transversely incised parenchymal tissue, based on data from the surgical stapling and cutting instruments, including data from their cartridges. The cartridge data may correspond, for example, to the size or type of staples fired by the instruments. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hubs 106, 206 to infer that the segmental resection portion of the procedure has been performed.

[0149] Next, in step 12, step 2524, the nodular dissection step is performed. Based on the data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hubs 106, 206 can infer that the surgical team is dissecting the node and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely dissected corresponds to the nodular dissection step, thereby enabling the surgical hubs 106, 206 to make the above inference. It should be noted that surgeons periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Furthermore, in certain examples, robotic tools may be used in one or more steps of the surgical procedure and / or handheld surgical instruments may be used in one or more steps of the surgical procedure. The surgeon(s) may use, for example, robotic tools and handheld surgical instruments alternately, and / or simultaneously. Once step 12(2524) is completed, the incision is closed and the postoperative portion of the procedure begins.

[0150] In the 13th step 2526, the patient is recovered from anesthesia. The surgical hubs 106, 206 can estimate that the patient is waking from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase).

[0151] Finally, in step 14, 2528, the medical personnel remove various patient monitoring devices from the patient. Thus, the surgical hubs 2106, 2236 can infer that the patient has been transferred to the recovery room when the hubs lose EKG, BP, and other data from the patient monitoring devices. As can be seen from this exemplary procedure description, the surgical hubs 2106, 2236 can determine or infer when each step of a given surgical procedure is being performed based on data received from various data sources that are communicably connected to the surgical hubs 2106, 2236.

[0152] Situational awareness is further described in various U.S. patent applications incorporated herein by reference, which are incorporated herein by reference in their entirety. In certain examples, the operation of a robotic surgical system, including, for example, various robotic surgical systems disclosed herein, may be controlled by hubs 2106, 2236 based on its situational awareness and / or feedback from its components, and / or information from cloud 2104 (Figure 17).

[0153] Analysis of surgical procedure trends and surgical techniques One problem inherent in surgical procedures is that they are performed by individuals who may utilize different techniques when performing any given surgical procedure. In some cases, the surgical outcome associated with any given determinant in a surgical procedure may be direct and easily identifiable. For example, the amount of bleeding that occurs after an incision is generally directly and easily identifiable because a person can visualize blood, and it correlates highly with the act of making the incision. However, in many cases, the surgical outcome associated with the determinant of a surgical procedure can be significantly attenuated from the determinant itself. For example, there may be a significant time delay (e.g., several years) in the readmission of a patient who has undergone a surgical procedure performed by a given surgeon, and it is unlikely that it will be possible to determine whether a particular measure taken by the surgeon led to the readmission. This dynamism can create a substantial disconnect between identifying and correcting surgical techniques that are not ideal or otherwise not associated with the most positive surgical outcomes. However, the surgical system may be configured to track perioperative surgical data via surgical hubs 2106, 2236, etc., as described above under the heading SURGICAL HUB SYSTEM, for analysis by a computer system (e.g., cloud 2204 and / or remove server 2213). Furthermore, the surgical system may be configured to determine contextual information associated with surgical procedures, as described above under the heading SITUATIONAL AWARENESS. Leveraging the ability to track perioperative surgical data, determine contextual information associated with surgical procedures, and analyze all of this data across a network of surgical systems spanning regional or even globally, it is possible to identify and track trends associated with various decision points related to surgical procedures (e.g., the type of surgical instrument used in the procedure, the location of the incision, or the amount of tissue removed), and then determine the measures that are most highly correlated with positive surgical outcomes (e.g., the amount of bleeding at the incision, whether any intraoperative corrective measures were needed, reoperation rate, postoperative bleeding rate, or readmission rate) to propose specific measures at various decision points related to surgical procedures.

[0154] In various embodiments, a surgical system may be configured to monitor actions taken by a user when performing a surgical procedure and then provide recommendations or alerts when actions deviate from a baseline. The baseline may be determined by monitoring or recording the performance of surgical procedures, determining surgical outcomes associated with various surgical procedures, determining which particular surgical actions are most associated with positive surgical outcomes, and then establishing a baseline for various surgical actions in each surgical procedure type according to which actions are associated with positive surgical outcomes. For example, Figure 22 is a diagram of a surgical system 7000 that can be configured to implement the techniques described above. The surgical system 7000 includes a control system 7002 connected to an imaging system 7004 and a backend computer system 7010 via a data network (e.g., LAN, WAN, or the Internet). In one embodiment, the control system 7002 may include a control system 133 as described in relation to Figure 2, surgical hubs 2106, 2236 as described in relation to Figures 17-19, and other such systems. The control system 7002 may include the imaging system 142 (Figure 2), or any other such imaging or visualization system described in relation to Figures 1 to 19, which may include imaging systems configured to utilize structural electromagnetic radiation (EMR) techniques and / or multispectral imaging techniques to characterize objects. The backend computer system 7010 may include a cloud computing architecture or another computer system configured to store and execute various machine learning models or other algorithms.

[0155] By visualizing, at least partially, what is happening during a surgical procedure via the imaging system 7004, the surgical system 7000 can monitor decision points within the procedure (e.g., instrument selection, stapler cartridge selection, or sequence of surgical steps) and log these decisions. The surgical system 7000 can monitor intraoperative decision points by utilizing the imaging system 7004 to visualize objects within the field of view (FOV) of the imaging system 7004, and then by performing object recognition or other computer vision techniques (e.g., by the control system 7002) to identify surgical instruments used in the surgical procedure, specific organs or tissues being operated on, etc. The identified actions taken by the surgeon at various decision points can then be used to inform an algorithm that balances patient factors, surgeon factors, instrument usage, and clinical outcome data, and to train a machine learning model (e.g., an artificial neural network) using, for example, supervised or unsupervised machine learning techniques. Once trained, the machine learning model can provide suggestions to the user when statistically significant outcomes may be influenced by decision points during the surgical procedure. Furthermore, these machine learning models or other algorithms can be used to postoperatively review actions taken by surgical staff during surgical procedures and flag actions for review by the surgical staff. The surgical staff can then be given the opportunity to approve or disagree with each flagged evaluation, thereby better informing and training both the surgical staff and the algorithms. In one embodiment, the control system 7002 may be configured to collect perioperative data and then provide the data (e.g., intraoperatively or postoperatively) to a backend computer system 7010 (e.g., a cloud computing system) via a data network 7008. The backend computer system 7010 may then be configured to run and train machine learning models or other algorithms based on the data provided by the connected control systems 7002 or the network of control systems 7002.In one embodiment, the trained machine learning model is run by the backend computer system 7010 and can provide real-time recommendations or analyses to the control system 7002 during the performance of a surgical procedure, based on intraoperative data provided by the control system 7002. In another embodiment, the trained machine learning model can be provided to and run by the control system 7002 itself.

[0156] In one embodiment, the surgical system 7000 may be configured to identify objects within the FOV of the imaging system 7004 by analyzing various actions taken during a surgical procedure, such as the type of surgical instrument selected to perform a given surgical procedure step, or the position or orientation of the surgical instrument relative to the patient's tissue, via an imaging system 7004 including a structural optics system (e.g., structural light source 152), thereby enabling adaptive responses and comparisons between current and previous actions in similar surgical procedure types. In particular, the surgical system 7000 may be configured to compile perioperative data from multiple data sources to provide structural optics tracking trends and baselines for objects during a surgical procedure. The visualized surgical procedure data can be compared with clinical outcomes occurring during or after the procedure to determine trends in techniques used for a particular surgical procedure step, types of surgical instruments used, and other determinants with clinical outcomes that provide future baselines. Thus, these baselines can define best practices for performing a given surgical procedure.

[0157] In one implementation, the control system 7002 may be configured to execute various control algorithms, including surface mapping logic 136, imaging logic 138, tissue identification logic 140, distance determination logic 141, targeting logic, and / or trajectory projection logic, as described in relation to Figure 2. The control algorithms may be configured to control surgical instruments or other components of the surgical system 7000, display information to the user, and / or control the robotic system 2110 (Figure 17). The various control algorithms executed by the control system 7002 can be further improved by utilizing machine learning models or algorithms trained on perioperative surgical motion data and surgical outcome data. In particular, the targeting logic or trajectory projection logic can be refined based on visualization of tissue volume and surface (for example, as determined by the structural optics system of the imaging system 7004) in combination with locally measured surgical instrument parameters. This intraoperative data can then be used in conjunction with surgical outcome data, which may include data related to the interaction between tissue and surgical instruments, to improve the control algorithms of the device.

[0158] In another implementation, the control system 7002 may be configured to record the position of surgical instruments used in surgical procedures on a patient in a local coordinate system or a global coordinate system. The local coordinate system may be defined, for example, with respect to the surgical instrument itself, a specific tissue or organ, or a virtual viewpoint, as described in the concurrently filed patent application entitled "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM," with agent reference number END9227USNP1 / 190579-1, which is incorporated herein by reference in its entirety. The global coordinate system may be defined, for example, with respect to the patient or the operating room. In combination with recording the position of the surgical instrument, the control system 7002 may be configured to record functions performed by the surgical instrument, such as whether the surgical instrument has been fired, the power level of the surgical instrument, or tissue characteristics or other parameters sensed by the surgical instrument. Accordingly, the position and function of surgical instruments can be used to inform / train machine learning models or other algorithms, thereby correlating the relative position and function of surgical instruments with positive surgical outcomes, which can help surgeons improve their skills or provide more accurate information for studying and improving surgical instrument functions. For example, an algorithm could be developed to modify the function of joint movement buttons on surgical instruments based on data showing that modifying the function of joint movement buttons leads to better surgical outcomes, possibly because the joint movement buttons are not intuitive when the surgical instrument is in a particular position and / or when performing a particular function.

[0159] In another implementation, the control system 7002 may be configured to continuously monitor the surgeon's movements, looking for unnecessary steps, unnecessary tissue contact / manipulation, or unexpected actions / steps based on situational awareness. The control system 7002 can better inform / train both the surgeon and the algorithm by flagging such identified events for postoperative review by the surgeon and providing the surgeon with the opportunity to approve or disagree with each flagged evaluation.

[0160] An example of an algorithm that can be used to implement the various technologies or implementations described above is shown in Figure 23, which is a logical flow diagram of process 7100 for providing dynamic surgical recommendations to the user. In the following description of process 7100, please also refer to Figures 2, 17-19 and 22. Process 7100 may be embodied, for example, as instructions stored in memory 134 connected to control circuit 132, which, when executed by control circuit 132, cause control circuit 132 to perform the enumerated steps of process 7100. It should be understood that process 7100 may be executed by and / or between control system 7002 (which may include surgical hubs 2106, 2236) and backend computer system 7010 (which may include cloud 2204 or remote server 2213). Accordingly, control circuit 132 may collectively refer to one or more control circuits associated with or distributed between control system 7002 and backend computer system 7010. In other words, the control circuit 132 may include the control circuit of the control system 7002 and / or the control circuit of the backend computer system 7010. For brevity, process 7100 is described as being performed by the control system 7002 and the backend computer system 7010. However, it should be understood that process 7100 may be performed by other combinations of hardware, software, and / or firmware, and any particular step of process 7100 may be performed by either the control system 7002 or the backend computer system 7010.

[0161] Accordingly, the control system 7002 and / or backend computer system 7010 that executes process 7100 may receive images and perioperative data of the surgical procedure being performed (e.g., acquired via the imaging system 7004). The images may be associated with perioperative data such as the position of the surgical instrument in a local or global coordinate system, sensor measurements, and object recognition data. Furthermore, the images may be generated at least in part based on a structural optical system, and thus may include three-dimensional (3D) volumetric data or surface mapping data. In one implementation, the control system 7002 may first generate images via the imaging system 7004 and then provide the image data to the backend computer system 7010 for processing. The backend computer system 7010 may be communicably connected to several different control systems 7002, which may then be located in or associated with several different facilities or hospital networks. Thus, the backend computer system 7010 can receive surgical image data across several different control systems 7002 to train machine learning models or other algorithms.

[0162] Accordingly, the control system 7002 and / or the backend computer system 7010 can determine the surgical outcome associated with the surgical procedure for which perioperative data have been received 7104. In one embodiment, the control system 7002 can determine the surgical outcome by, for example, visualizing the surgical outcome (e.g., bleeding along the incision line) via the imaging system 7004. In another embodiment, the backend computer system 7010 can determine the surgical outcome by, for example, storing a database of surgical outcomes associated with a given surgical procedure. The database may be updated when additional patient-related information is received by the backend computer system 7010, or the backend computer system 7010 may allow a user to update the database when a surgical outcome is identified. For example, a user (e.g., a healthcare facility staff member) may update the database when a patient returns for a reoperation or when an excessive amount of pain or other negative outcome associated with a surgical procedure is reported.

[0163] Accordingly, the control system 7002 and / or the backend computer system 7010 can determine baseline surgical actions corresponding to image and outcome data 7106. In one embodiment, the backend computer system 7010 can be programmed to run and train a machine learning model to correlate received images and other perioperative data with determined outcomes to establish surgical actions at each defined decision point in the surgical procedure that is most correlated (or at least correlates above a threshold) to a desired or positive surgical outcome. Thus, such surgical actions can be defined as baseline or recommended surgical actions performed at each decision point associated with each surgical procedure type. Once trained, the machine learning model can then be used to provide preoperative, intraoperative, or postoperative recommendations to the user according to the defined baseline.

[0164] Accordingly, the control system 7002 may generate images of the surgical site during a surgical procedure, for example, via the imaging system 7004 7108. The images may be generated using structured EMR, multispectral imaging, or any other imaging technique described above 7108. Furthermore, the images may be associated with various perioperative data, including, for example, surface mapping or 3D geometric shape determined via structured EMR, subsurface or tissue characteristic data determined via multispectral imaging techniques, object recognition data, positional data relative to a global coordinate system and / or local coordinate system, and / or contextual data determined via a situation recognition system.

[0165] Accordingly, the control system 7002 and / or the backend computer system 7010 can determine what surgical action is being performed at a given decision point in a surgical procedure. In one embodiment, the control system 7002 and / or the backend computer system 7010 may make this determination via a situational awareness system. For example, as described above in relation to Figure 21, the control system 7002 can determine whether the surgeon is making an incision to move the lung, ligating a blood vessel, or transversely incising parenchymal tissue, based on generator data and surgical instrument data (including the relative operation of the surgical device). In this embodiment, the decision point of the surgical action includes which device is being used for a particular step of the surgical procedure (e.g., an ultrasonic device or an electrosurgical device, the size and type of a staple cartridge, or the brand of a surgical instrument), where or what the surgeon is transversely incising or ligating, etc. In another embodiment, the control system 7002 and / or the backend computer system 7010 can determine what preoperative mixture of surgical instruments is planned for a surgical procedure by visualizing the preparation table in the operating room. In this embodiment, the determining factor for the surgical action is therefore the surgical instrument selected for the procedure.

[0166] Accordingly, the control system 7002 and / or the backend computer system 7010 may compare the current surgical action with a baseline surgical action with respect to a given decision point of the surgical procedure determined using the techniques described above 7112. Accordingly, the control system 7002 and / or the backend computer system 7010 may provide recommendations based on the comparison between the surgical action and the baseline 7114. In various embodiments, recommendations may be provided preoperatively (e.g., recommending different mixtures of surgical instruments for performing the procedure), intraoperatively (e.g., recommending different positions of the end effectors of surgical instruments), or postoperatively (e.g., recommending different surgical actions at flagged points during the procedure via a report reviewed by the surgeon). In one embodiment, recommendations may be provided only when the surgical action deviates from the baseline by at least a threshold amount. Recommendations may take the form of audible alerts, text or graphic feedback, haptic feedback, etc. In one embodiment shown in Figure 24, a display 7006 connected to a control system 7002 may display a video feed 7200 of a surgical site 7014 provided by an imaging system 7004. The video feed 7200 may show the position of a surgical instrument 7210, as well as visualization of other tissues and / or structures, including a tumor 7016 and various blood vessels 7018 in this particular embodiment. In this embodiment, the control system 7002 and / or the backend computer system 7010 have determined that the position of the surgical instrument 7210 has deviated from the baseline position for a given step of the surgical procedure. Accordingly, the control system 7002 provides the display 7006 with a recommendation in the form of a graphical overlay 7212 indicating a recommended or baseline position for the surgical instrument 7210 for a given step of the surgical procedure 7114.

[0167] These systems and methods enable surgeons to visualize the recommended action sequence for any given decision point in a surgical procedure and act accordingly. Thus, surgeons can learn and further improve based on feedback derived from massive amounts of data collected from any number of surgical procedures performed by any number of individuals worldwide. This allows surgeons to further refine intraoperative techniques or other decisions associated with the performance of surgical procedures in order to improve patient outcomes. Furthermore, such systems can be integrated into robotic surgical systems to leverage vast amounts of available surgical data to improve their control algorithms and thereby control the operation of the robotic surgical system over time.

[0168] In one embodiment, the surgical system can also be configured to predict and project the effects of actions that may be taken for the user during a surgical procedure. In particular, 3D surface and volume visualization of tissues and / or structures can be combined with computational analysis and modeling datasets derived from data sources to record the effects of various surgical devices on tissues and / or structures. For example, the surgical system 7000 can visualize the surgical site 7014 (including any tissues and / or structures located therein) via the imaging system 7004, and then record the effects on various tissues and / or structures in response to various treatments, such as firing a surgical stapler or an electrosurgical instrument. The effects on tissues and / or structures can then be recorded and modeled for any given type, size, and configuration of the tissues and / or structures. Furthermore, computer analysis can then be performed (e.g., by the control system 7002) to predict the nature of the therapeutic projection from the surgical devices used in the surgical procedure. For example, the therapeutic projection may include the application of thermal or electrical energy to the tissue, and the extent to which the energy application penetrates the 3D scanned volume of the tissue. This therapeutic projection can be provided to the user, for example, as a graphic overlay on a video feed 7200 shown on display 7006.

[0169] In one embodiment, projections from a model can be determined and compared separately from predictions generated by other models or surgical devices. For example, the imaging system 7004 may include a secondary sensing array (e.g., an IR CMOS array) configured to sense tissue features, project the application of energy to the tissue, and compare the projection with predicted projections from other models or surgical devices (e.g., an electrosurgical generator). Furthermore, predictions from other data sources can be adjusted based on the comparative accuracy of the projections modeled by the control system 7002 with respect to predictions from other data sources, so that they are determined according to surgical procedure history data. For example, visualization of the power levels of electrosurgical or ultrasonic surgical instruments and the interaction between tissue and the jaws of the instrument can be used to provide algorithms (e.g., based on finite element analysis) and boundary conditions for predictions (e.g., from a generator).

[0170] In one embodiment, when the prediction from the computer analysis of the imaging system 7004 differs from the prediction from an algorithm performed by other data (e.g., a surgical generator) by at least a threshold, the control system 7002 can perform a variety of different actions. For example, the control system 7002 can provide a user alert before the user opens the jaws to examine the tissue (e.g., emit an unusual sound), notify the user after the completion of a surgical step (e.g., emit a different tone when energy delivery by the surgical instrument has ended), soft lock out the jaws of the surgical instrument (until disabled by the user), or adjust the generator control algorithm to adjust energy delivery to the tissue (e.g., extend energy delivery to ensure a safe outcome at the expense of additional time).

[0171] These systems and methods enable surgeons to visualize the predicted motion of any given decision point in a surgical procedure and act accordingly. Therefore, surgeons can make more informed intraoperative decisions based on feedback derived from a vast amount of data collected from any number of surgical procedures performed by any number of individuals worldwide. This allows surgeons to further refine their intraoperative techniques to improve patient outcomes. Furthermore, such systems can be integrated into robotic surgical systems to leverage the enormous amount of available surgical data to improve their control algorithms and thereby control the operation of the robotic surgical system over time.

[0172] Exemplary clinical uses The various surgical visualization systems disclosed herein may be used in one or more of the following clinical applications. These clinical applications are not exhaustive and are merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0173] The surgical visualization system, as disclosed herein, can be used for many different types of procedures in different medical specialties, such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, obesity / gastric surgery, and hepatobiliary and pancreatic surgery (HPB). For example, in urological surgery such as prostatectomy, the urinary tract may be detected in fat or connective tissue, and / or nerves may be detected, for example, in fat. For example, in gynecological oncology surgery such as hysterectomy, and in colorectal surgery such as low anterior resection (LAR), the ureters may be detected, for example, in fat and / or connective tissue. For example, in thoracic surgery such as lobectomy, blood vessels may be detected in the lungs or connective tissue, and / or nerves may be detected in connective tissue (e.g., esophagostomy). In bariatric surgery, blood vessels may be detected in fat. For example, in HPB surgeries such as hepatectomy or pancreatectomy, blood vessels may be detected in fatty tissue (extrahepatic) or connective tissue (extrahepatic), and bile ducts may be detected in parenchymal tissue (liver or pancreas).

[0174] In one embodiment, a clinician may desire the removal of an endometrial fibroid. From a preoperative magnetic resonance imaging (MRI) scan, the clinician can determine that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may desire to know during surgery which tissues constitute part of the intestine and which tissues constitute part of the rectum. In such an example, a surgical visualization system, as disclosed herein, can display different types of tissue (intestine vs. rectum) and transmit this information to the clinician via the imaging system. Furthermore, the imaging system can determine and communicate the proximity of surgical instruments to the selected tissues. In such an example, the surgical visualization system can improve the efficiency of the procedure without serious complications.

[0175] In another embodiment, a clinician (e.g., a gynecologist) may remain at a distance from certain anatomical regions to avoid getting too close to critical structures, thus potentially preventing the removal of, for example, all of the endometriosis. A surgical visualization system, as disclosed herein, can reduce the risk of a gynecologist getting too close to critical structures, thereby allowing the gynecologist to get close enough to remove all of the endometriosis using surgical instruments, thereby improving patient outcomes (democratizing surgery). Such a system can enable a surgeon to "keep moving" during a surgical procedure instead of repeatedly stopping and restarting to identify areas to avoid, for example, during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine arteries and ureters are critical structures, and the system may be particularly useful in hysterectomies and endometrial surgeries, given the presentation and / or thickness of the tissues involved.

[0176] In another embodiment, clinicians may risk dissecting a vessel in a location that is too close and could affect blood supply to lobes other than the target lobe. Furthermore, anatomical differences between patients may result in dissecting a vessel (e.g., a branching vessel) that affects different lobes based on a particular patient. A surgical visualization system, as disclosed herein, can enable the identification of the correct vessel at the desired location, thereby enabling clinicians to dissect with appropriate anatomical confidence. For example, the system can verify that the correct vessel is in the correct location, after which the clinician can safely divide the vessel.

[0177] In another embodiment, clinicians may make multiple incisions before finding the best location for incision due to uncertainty regarding the anatomical structure of the blood vessels. However, since more incisions can increase the risk of bleeding, it is desirable to first incise the location. Surgical visualization systems, as disclosed herein, can minimize the number of incisions by indicating the correct blood vessel and the best location for incision. For example, the ureters and cardinal ligaments are densely packed and present unique challenges during incision. In such cases, minimizing the number of incisions may be particularly desirable.

[0178] In another embodiment, a clinician removing cancerous tissue (e.g., a surgical oncologist) may want to know how to identify important structures, locate the cancer, stage the cancer, and / or assess the normality of the tissue. Such information goes beyond what the clinician sees with the "naked eye." Surgical visualization systems, as disclosed herein, can determine and / or communicate such information to the clinician during surgery, thereby enhancing intraoperative decision-making and improving surgical outcomes. In certain examples, surgical visualization systems may be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches using either an endoscope or an exoscopy, for example.

[0179] In another embodiment, a clinician (e.g., a surgical oncologist) may want to turn off one or more alerts regarding the proximity of a surgical tool to one or more critical structures in order to avoid becoming overly conservative during a surgical procedure. In another example, a clinician may want to receive certain types of alerts, such as tactile feedback (e.g., vibration / buzzer), to indicate proximity and / or “no-fly zones” in order to stay sufficiently far from one or more critical structures. The surgical visualization system can provide adaptability based, for example, on the clinician’s experience and / or the desired aggressiveness of the procedure, as disclosed herein. In such an example, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. The surgical visualization system can assist in planning the next steps during a surgical procedure.

[0180] Various aspects of the subject matter described herein are illustrated in the following numbered examples.

[0181] Example 1. A surgical control system that is communicably connected to a backend computer system, the surgical control system comprising an imaging system and a control circuit connected to the imaging system. The imaging system comprises an emitter configured to emit electromagnetic radiation (EMR) and an image sensor configured to receive EMR reflected from the surgical site. At least a portion of the EMR is emitted as structured EMR. The control circuit is configured to generate an image of the surgical site via the reflected EMR received by the image sensor, to determine the surgical action being performed based on the image, to receive a baseline surgical action associated with the surgical action from the backend computer system, and to provide user recommendations according to a comparison between the surgical action and the baseline surgical action.

[0182] Example 2. The surgical control system according to Example 1, wherein the surgical control system is communicably connected to a display screen, and user recommendations include a graphical display of baseline surgical movements for surgical movements displayed on the display screen.

[0183] Example 3. The surgical control system according to Example 1 or 2, wherein the control circuit is configured to provide user recommendations based on whether the comparison between the surgical action and the baseline surgical action differs by at least a threshold.

[0184] Example 4. A surgical control system according to any one of Examples 1 to 3, wherein the control circuit is further configured to receive perioperative data and to determine the surgical action being performed based on the image and the received perioperative data.

[0185] Example 5. The surgical control system according to Example 4, wherein the perioperative data includes at least one of surgical generator data or surgical instrument sensor data.

[0186] Example 6. A surgical control system according to any one of Examples 1 to 5, wherein a surgical hub, which is communicably connected to one or more surgical devices, comprises a surgical control system.

[0187] Example 7. A surgical control system according to any one of Examples 1 to 5, wherein the surgical control system is communicably connectable to a surgical hub, and the control circuit is configured to determine surgical actions being performed based on images according to an analysis algorithm, and to receive updates to the analysis algorithm from the surgical hub.

[0188] Example 8. A computer system that can communicate with a plurality of surgical hubs, each of which can communicate with an imaging system, wherein the computer system comprises a control circuit configured to receive from the plurality of surgical hubs a plurality of images of a plurality of surgical sites acquired by each imaging system during a plurality of surgical procedures, and to determine a plurality of surgical outcomes, wherein each of the plurality of surgical outcomes is associated with one of the plurality of surgical procedures and determined based on one of the plurality of images, to determine a baseline surgical action according to the plurality of images and the plurality of surgical outcomes, and to transmit the baseline surgical action to the plurality of surgical hubs.

[0189] Example 9. The computer system according to Example 8, wherein the control circuit is configured to train a machine learning model to determine the baseline surgical action of each surgical action by correlating surgical actions with positive surgical outcomes based on multiple images and multiple surgical outcomes.

[0190] Example 10. The computer system according to Example 8 or 9, wherein the control circuit is configured to receive perioperative data associated with multiple surgical procedures from multiple surgical hubs and to determine multiple surgical outcomes based on multiple images and perioperative data.

[0191] Example 11. The computer system according to any one of Examples 8 to 10, wherein the control circuit is configured to receive multiple surgical outcomes as user input.

[0192] Example 12. A method for controlling a surgical system that is communicably connected to a backend computer system, the control system comprising an imaging system, the imaging system comprising an emitter configured to emit electromagnetic radiation (EMR), wherein at least a portion of the EMR is emitted as structured EMR, and an image sensor configured to receive EMR reflected from a surgical site, the method comprising: generating an image of the surgical site via reflected EMR received by the image sensor; determining a surgical action being performed based on the image; receiving a baseline surgical action associated with the surgical action from a backend computer system; and providing user recommendations according to a comparison between the surgical action and the baseline surgical action.

[0193] Example 13. The method according to Example 12, wherein the surgical system includes a display screen, and user recommendations include a graphical display of baseline surgical movements for surgical movements displayed on the display screen.

[0194] Example 14. The method according to Example 12 or 13, wherein user recommendations are provided according to whether the comparison between the surgical action and the baseline surgical action differs by at least a threshold.

[0195] Example 15. The method according to any one of Examples 12 to 14, further comprising receiving perioperative data and determining the surgical action being performed based on the images and the received perioperative data.

[0196] Example 16. The method according to Example 15, wherein the perioperative data includes at least one of surgical generator data or surgical instrument sensor data.

[0197] Example 17. The method according to any one of Examples 12 to 16, further comprising: a surgical system comprising a surgical hub; a method comprising determining a surgical action being performed based on an image according to an analysis algorithm; and receiving updates to the analysis algorithm from the surgical hub.

[0198] While several forms have been shown and described, it is not the applicant's intention to limit or restrict the attached claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be implemented and will be conceived by those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element related to the described form can be alternatively described as a means for providing the function performed by that element. Also, while a material is disclosed with respect to a particular component, other materials may be used. Therefore, it should be understood that the above specification and attached claims are intended to cover all such modifications, combinations, and variations as being included within the scope of the disclosed forms. The attached claims are intended to cover all such modifications, variations, alterations, substitutions, alterations, and equivalents.

[0199] The detailed descriptions above have used block diagrams, flowcharts, and / or examples to illustrate various forms of apparatus and / or processes. As far as such block diagrams, flowcharts, and / or examples include one or more functions and / or operations, it should be understood by those skilled in the art that each function and / or operation included in such block diagrams, flowcharts, and / or examples can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some of the forms disclosed herein, in whole or in part, can be equivalently implemented on an integrated circuit as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code falls within the scope of the skills of those skilled in the art in light of this disclosure. In addition, those skilled in the art will understand that the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and the specific forms of the subject matter described herein apply regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0200] Instructions used to program logic to implement various disclosed embodiments may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but are not limited to floppy diskettes, optical disks, compact disks, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage used for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, non-temporary computer-readable media can include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0201] When used in any aspect of this specification, the term “control circuit” can mean, for example, hardwired circuits, programmable circuits (e.g., computer processors, processing units, processors, microcontrollers, microcontroller units, controllers, digital signal processors (DSPs), programmable logic devices (PLDs), programmable logic arrays (PLAs), or field programmable gate arrays (FPGAs) including one or more individual instruction processing cores), state-machine circuits, firmware that stores instructions executed by programmable circuits, and any combination thereof. Control circuits can be embodied collectively or individually as circuits that form part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, or a smartphone. Accordingly, as used herein, “control circuit” includes, but is not limited to, an electrical circuit having at least one separate electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application-specific integrated circuit, an electrical circuit forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program that performs at least part of the processes and / or devices described herein, or a microprocessor configured by a computer program that performs at least part of the processes and / or devices described herein), an electrical circuit forming a memory device (e.g., in the form of random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electric installation).Those skilled in the art will recognize that the subject matter described herein may be implemented in analog form, digital form, or a combination thereof.

[0202] When used in any aspect of this specification, the term “logic” may mean an application, software, firmware, and / or circuit configured to perform any of the operations described above. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-temporary computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and / or hardcoded (e.g., non-volatile) data in a memory device.

[0203] When used in any aspect of this specification, terms such as “component,” “system,” and “module” may refer to computer-related entities that are hardware, a combination of hardware and software, software, or running software.

[0204] Where used in any aspect of this specification, “algorithm” means a self-consistent sequence of steps leading to a desired result, and “step” means the manipulation of physical quantities and / or logical states that can take the form of electrical or magnetic signals, which are not necessarily required but can be stored, transferred, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities, or simply are convenient labels applied to these quantities and / or states.

[0205] A packet-switched network is one example of a network. Communication devices can communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol is the Ethernet communication protocol, which can enable communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of the Ethernet standard. Alternatively or additionally, communication devices may communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices may communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol conforms to or may be compatible with standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol conforms to or may be compatible with the ATM standard and / or later versions of this standard, published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0".Naturally, different and / or later developed connection-type network communication protocols are equally construed herein.

[0206] Unless otherwise explicitly stated, as is evident from the foregoing disclosures, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the foregoing disclosures should be understood to refer to the operation and processing of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage, transmission, or display device.

[0207] One or more components may be referred to herein as “configured to,” “configurable to,” “operable / operative to,” “adapted / adaptable,” “able to,” “conformable / conformed to,” etc. Those skilled in the art will understand that “configured to” generally encompasses active components and / or inactive components and / or standby components, unless the context should interpret it otherwise.

[0208] The terms “proximal” and “distal” are used herein in reference to the clinician operating the handle portion of a surgical instrument. “Proximal” refers to the part closest to the clinician, and “distal” refers to the part further away from the clinician. For convenience and clarity, spatial terms such as “vertical,” “horizontal,” “up,” and “down” may be used herein in reference to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0209] Those skilled in the art will generally understand that the terms used herein, and especially in the appended claims (e.g., the text of the appended claims), are intended to be generally "open" terms (for example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "includes" should be interpreted as "includes but is not limited to"). Furthermore, those skilled in the art will understand that if a particular number is intended in an introduced claim recitation, such intention is clearly stated in the claim, and if such statement is not made, such intention does not exist. For example, to aid understanding, subsequent appended claims may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be interpreted as suggesting that any particular claim containing such introduced claim description is limited to claims containing only one such description, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" when the claim description is introduced by the indefinite article "a" or "an" (for example, "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using the definite article.

[0210] In addition, even if a specific number is explicitly stated in the introduced claim, it will be recognized by those skilled in the art that such a statement should typically be interpreted as meaning at least the number stated (for example, if there is a statement that is simply “two descriptions” without any other modifiers, it generally means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to “at least one of A, B, and C, etc.” is used, such a notation is generally intended to be understood in a way that those skilled in the art will understand (for example, “a system having at least one of A, B, and C” is not limited to systems having only A, only B, only C, both A and B, both A and C, both B and C and / or all of A, B and C, etc.). When expressions similar to "at least one of A, B, or C" are used, such expressions are generally intended to be understood in a way that a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Furthermore, a person skilled in the art will understand that typically any disjunctive word and / or phrase representing two or more selective terms should be understood, whether in the specification, claims, or drawings, as intended to include the possibility of including one of those terms, any of those terms, or both of those terms, unless the context requires a different interpretation. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0211] With respect to the attached claims, those skilled in the art will understand that the operations cited herein may generally be performed in any order. Furthermore, while flowcharts of various operations are shown in sequence, it should be understood that the various operations may be performed in an order other than those shown, or simultaneously. Examples of such alternative orderings may include repetition, alternation, interruption, reordering, augmentation, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context should interpret them otherwise. Moreover, terms such as “responding to,” “related to,” or other past tense adjectives are not generally intended to exclude such variations, unless the context should interpret them otherwise.

[0212] It is worth noting that any reference to “one aspect,” “aspect,” “example,” or “example” means that the specific mechanism, structure, or characteristic described in relation to that aspect is included in at least one aspect. Therefore, the phrases “in one aspect,” “in aspect,” “example,” and “example” found in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, specific features, structures, or characteristics can be combined in any preferred manner in one or more aspects.

[0213] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material does not conflict with this Specified. Any disclosure expressly stated herein, either in itself or to the extent required, shall supersede any conflicting statement incorporated herein by reference. Any material, or any portion thereof, that is referred to as being incorporated herein by reference but conflicts with current definitions, views, or other disclosures contained herein, shall be incorporated only to the extent that it does not create a conflict between the incorporated material and the current disclosures.

[0214] In summary, this specification describes the many benefits that can be obtained as a result of using the concepts described herein. The above descriptions of one or more forms are presented for illustrative and explanatory purposes only. They are not intended to be comprehensive or to be limited to the exact forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms are selected and described to illustrate the principle and practical applications, thereby enabling a person skilled in the art to utilize the various forms, along with various modifications, for specific conceivable uses. The claims presented with this specification are intended to define the overall scope.

[0215] [Implementation Method] (1) A surgical control system that can communicate with a backend computer system, wherein the surgical control system is An imaging system, An emitter configured to emit electromagnetic radiation (EMR), wherein at least a portion of the EMR is emitted as structured EMR, An imaging system comprising: an image sensor configured to receive the EMR reflected from the surgical site; A control circuit connected to the IM system, wherein the control circuit is The image of the surgical site is generated via the reflected EMR received by the image sensor, To determine the surgical procedure being performed based on the aforementioned image, Receiving a baseline surgical action associated with the surgical action from the aforementioned backend computer system, A surgical control system comprising a control circuit configured to provide user recommendations in accordance with a comparison between the aforementioned surgical action and the baseline surgical action. (2) The surgical control system is communicatively connected to a display screen, The surgical control system according to Embodiment 1, wherein the user recommendation includes a graphical representation of the baseline surgical action for the surgical action displayed on the display screen. (3) The surgical control system according to Embodiment 1, wherein the control circuit is configured to provide the user recommendation according to whether the comparison between the surgical action and the baseline surgical action differs by at least a threshold. (4) The control circuit, Receiving perioperative data, The surgical control system according to Embodiment 1, further configured to determine the surgical action being performed based on the aforementioned image and the received perioperative data. (5) The surgical control system according to Embodiment 4, wherein the perioperative data includes at least one of surgical generator data or surgical instrument sensor data.

[0216] (6) The surgical control system according to Embodiment 1, wherein a surgical hub, which is communicably connected to one or more surgical devices, comprises the surgical control system. (7) The surgical control system is connectable to a surgical hub in a communicative manner, The aforementioned control circuit Determining the surgical action being performed based on the image according to the analysis algorithm, A surgical control system according to Embodiment 1, configured to receive updates to the analysis algorithm from the surgical hub. (8) A computer system that can communicate with a plurality of surgical hubs, wherein each of the plurality of surgical hubs can communicate with an imaging system, and the computer system A control circuit, The plurality of surgical hubs receive multiple images of multiple surgical sites acquired by each imaging system during multiple surgical procedures, Determining a plurality of surgical outcomes, each of the plurality of surgical outcomes being associated with one of the plurality of surgical procedures and being determined based on one of the plurality of images Determining a baseline surgical action according to the plurality of images and the plurality of surgical outcomes A computer system comprising a control circuit configured to transmit the baseline surgical action to the plurality of surgical hubs (9) The computer system according to embodiment 8, wherein the control circuit is configured to train a machine learning model to correlate a surgical action with a positive surgical outcome based on the plurality of images and the plurality of surgical outcomes, and to determine the baseline surgical action of each surgical action (10) The control circuit Receiving perioperative data associated with the plurality of surgical procedures from the plurality of surgical hubs The computer system according to embodiment 8, configured to determine the plurality of surgical outcomes based on the plurality of images and the perioperative data

[0217] (11) The computer system according to embodiment 8, wherein the control circuit is configured to receive the plurality of surgical outcomes as an input by a user (12) A method of controlling a surgical system communicably connectable to a backend computer system, the control system comprising an imaging system, the imaging system comprising an emitter configured to emit electromagnetic radiation (EMR), at least a portion of the EMR being emitted as structured EMR, and an image sensor configured to receive the EMR reflected from the surgical site, the method comprising Generating an image of the surgical site via the reflected EMR received by the image sensor Determining a surgical action being performed based on the image Receiving a baseline surgical operation associated with the surgical operation from the backend computer system; Providing user recommendations according to a comparison between the surgical operation and the baseline surgical operation. A method comprising the above. (13) The surgical system comprises a display screen; The method according to embodiment 12, wherein the user recommendation includes a graphic display of the baseline surgical operation with respect to the surgical operation displayed on the display screen. (14) The method according to embodiment 12, wherein the user recommendation is provided according to whether the comparison between the surgical operation and the baseline surgical operation is different by at least a threshold value. (15) Receiving perioperative data; The method according to embodiment 12, further comprising determining a surgical operation being performed based on the image and the received perioperative data.

[0218] (16) The method according to embodiment 15, wherein the perioperative data includes at least one of surgical generator data or surgical instrument sensor data. (17) The surgical system comprises a surgical hub; The method comprises: Determining the surgical operation being performed based on the image according to an analysis algorithm; The method according to embodiment 12, further comprising receiving an update for the analysis algorithm from the surgical hub.

Claims

1. A surgical control system that can communicate with a backend computer system, wherein the surgical control system is An imaging system, An emitter configured to emit electromagnetic radiation (EMR), wherein at least a portion of the EMR is emitted as structured EMR comprising a pattern of light including at least one of stripes, grid lines, and / or dots; An imaging system comprising: an image sensor configured to receive reflected EMR reflected from the surgical site; A control circuit connected to the IM system, wherein the control circuit is The image of the surgical site is generated via the reflected EMR received by the image sensor, To determine the surgical procedure being performed based on the aforementioned image, Receiving a baseline surgical action associated with the surgical action from the aforementioned backend computer system, A control circuit is configured to provide user recommendations in accordance with a comparison between the surgical procedure and the baseline surgical procedure, The emitter and the image sensor are located on the imaging system positioned within the patient's cavity. The EMR and the reflected EMR are near-infrared (NIR) or short-wave infrared (SWIR), A surgical control system wherein the control circuit is configured to determine the distance between the emitter and the image sensor and the surgical site based on the delay between the emitted EMR and the received reflected EMR.

2. The surgical control system is communicatively connected to a display screen, The surgical control system according to claim 1, wherein the user recommendation includes a graphical representation of the baseline surgical action for the surgical action displayed on the display screen.

3. The surgical control system according to claim 1, wherein the control circuit is configured to provide the user recommendation based on whether the comparison between the surgical action and the baseline surgical action differs by at least a threshold.

4. The surgical control system according to claim 1, wherein a surgical hub, which is communicably connected to one or more surgical devices, comprises the surgical control system.

5. The aforementioned control circuit Receiving perioperative data, The system is further configured to determine the surgical action being performed based on the aforementioned images and the received perioperative data, The surgical control system according to claim 4, wherein the perioperative data includes at least one of data from a surgical generator that drives the surgical device, or data from a sensor configured to detect parameters associated with the patient and / or the surgical device.

6. The surgical control system is connectable to a surgical hub in a communicative manner. The aforementioned control circuit Determining the surgical action being performed based on the image according to the analysis algorithm, The surgical control system according to claim 1, configured to receive updates to the analysis algorithm from the surgical hub.

7. A method for controlling a surgical control system that is communicably connected to a backend computer system, wherein the surgical control system comprises an imaging system and a control circuit connected to the imaging system, the imaging system comprising an emitter configured to emit electromagnetic radiation (EMR), wherein at least a portion of the EMR is emitted as structured EMR comprising a pattern of light including at least one of stripes, grid lines, and / or dots, and an image sensor configured to receive reflected EMR reflected from a surgical site, wherein the method is The image of the surgical site is generated via the reflected EMR received by the image sensor, To determine the surgical procedure being performed based on the aforementioned image, Receiving a baseline surgical action associated with the surgical action from the aforementioned backend computer system, To provide user recommendations based on a comparison between the aforementioned surgical procedure and the baseline surgical procedure, The emitter and the image sensor are located on the imaging system positioned within the patient's cavity. The EMR and the reflected EMR are near-infrared (NIR) or short-wave infrared (SWIR), A method comprising the control circuit determining the distance between the emitter and the image sensor and the surgical site based on the delay between the emitted EMR and the received reflected EMR.

8. The surgical control system includes a display screen, The method according to claim 7, wherein the user recommendation includes a graphical representation of the baseline surgical action for the surgical action displayed on the display screen.

9. The method according to claim 7, wherein the user recommendation is provided according to whether the comparison between the surgical action and the baseline surgical action differs by at least a threshold.

10. The surgical control system includes a surgical hub, The method described above is Determining the surgical action being performed based on the image according to the analysis algorithm, The method according to claim 7, further comprising receiving updates to the analysis algorithm from the surgical hub.

11. Receiving perioperative data, The method further includes determining the surgical action being performed based on the aforementioned images and the received perioperative data, The method according to claim 10, wherein the perioperative data includes at least one of the following: data from a surgical generator that drives one or more surgical devices communicably connected to the surgical hub, or data from a sensor configured to detect parameters associated with the patient and / or the one or more surgical devices.

Images