System and method for changing the display overlay of the surgical field based on trigger events

Abstract

A surgical system is disclosed that includes an imaging device, a display configured to show a live stream of a surgical field of a surgical procedure, and a control system operably coupled to the imaging device and the display, wherein the live stream is captured by the imaging device, and the control system is configured to overlay information associated with the surgical procedure on the live stream, detect the occurrence of a trigger event, and adjust the overlaid information based on the occurrence of the trigger event.

Description

[[Technical Field]]

[0001] (Cross-Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 174,674 entitled "HEADS UP DISPLAY" filed on April 14, 2021, and U.S. Provisional Patent Application No. 63 / 284,326 entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS" filed on November 30, 2021, under 35 U.S.C. § 119(e), and the disclosures of each of these are incorporated herein by reference in their entireties. [[Background Art]]

[0002] The present disclosure relates to apparatuses, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. It would be desirable to provide an augmented reality interactive experience of a real-world environment where objects present in the real world are enhanced by overlaying computer-generated perceptual information, sometimes across multiple sensory modalities including visual, auditory, haptic, somatosensory, and olfactory, during a surgical procedure. In the context of the present disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, haptic, somatosensory, olfactory, or other sensory information onto the real-world image of the surgical field and instruments or other objects appearing in the surgical field. The images may be streamed in real time or may be still images.

[0003] Real-world surgical instruments include a variety of surgical devices, including energy, staplers, or combinations of energy and staplers. Energy-based medical devices include, but are not limited to, radio frequency (RF) based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including obesity, thoracic, colorectal, obstetric and gynecological, urological, and general surgery. [Overview of the project] [Means for solving the problem]

[0004] In various examples, the Disclosure provides a surgical system comprising an imaging device, a display configured to show a live stream of the surgical field of a surgical procedure, and a control system operably coupled to the imaging device and the display. The live stream is captured by the imaging device. The control system is configured to overlay information associated with the surgical procedure onto the live stream, detect the occurrence of trigger events, and adjust the overlaid information based on the occurrence of trigger events.

[0005] In various examples, the Disclosure provides a surgical system comprising an imaging device, a display configured to show a live stream of the surgical field of a surgical procedure, and a control system operably coupled to the imaging device and the display. The live stream is captured by the imaging device. The control system is configured to overlay information associated with the surgical procedure onto the live stream, set a trigger event count, detect partial trigger events, adjust the trigger event count based on the occurrence of partial trigger events, and adjust the overlaid information based on whether the trigger event count reaches or exceeds a trigger event threshold.

[0006] In various examples, the Disclosure provides a surgical system comprising an imaging device, a display configured to show a live stream of the surgical field of a surgical procedure, a situational awareness module, and a control system operably coupled to the imaging device, the display, and the situational awareness module. The live stream is captured by the imaging device. The control system overlays information associated with the surgical procedure onto the live stream, and the situational awareness module is configured to determine the steps of the surgical procedure, detect the occurrence of trigger events, and adjust the overlaid information based on the occurrence of trigger events and the steps of the surgical procedure. [Brief explanation of the drawing]

[0007] The various embodiments described herein with respect to both configuration and operation methods, along with their further purposes and advantages, can be best understood by referring to the following description in conjunction with the accompanying drawings. [Figure 1] This is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 2] This is a diagram of a surgical system used to perform surgical procedures in an operating room, according to one aspect of the present disclosure. [Figure 3] One aspect of the present disclosure is a visualization system, a robotic system, and a surgical hub paired with an intelligent instrument. [Figure 4] This figure shows a surgical data network, according to one aspect of the present disclosure, which includes a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This figure shows a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 6] This figure shows a surgical hub, including a plurality of modules connected to a modular control tower, according to one aspect of the present disclosure. [Figure 7] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 8] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 9] This figure shows an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. [Figure 10] This figure shows a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. [Figure 11] This figure shows a timeline of a situational awareness surgical procedure according to one aspect of the present disclosure. [Figure 12] This figure shows a surgical visualization system according to at least one aspect of the present disclosure. [Figure 13] This figure shows a method for determining the display arrangement of competing surgical data for presentation on a display showing a live stream of the surgical field, according to at least one aspect of the present disclosure. [Figure 14] This figure shows a method for determining the display arrangement of competing surgical data for presentation on a display showing a live stream of the surgical field, according to at least one aspect of the present disclosure. [Figure 15] This figure shows a method for determining the display arrangement of competing surgical data for presentation on a display showing a live stream of the surgical field, according to at least one aspect of the present disclosure.

[0008] Throughout the drawings, corresponding reference numerals indicate corresponding parts. The examples described herein illustrate various disclosed embodiments in one form, and such examples should not be construed as limiting the scope thereof. [Modes for carrying out the invention]

[0009] The applicant of this application owns the following concurrently filed U.S. patent applications, the entirety of which is incorporated herein by reference: • U.S. Patent Application titled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Agent Reference Number END9352USNP1 / 210120-1M • U.S. Patent Application titled "UTILIZATION OF SURGICAL DATA VALUES AND SITUATIONAL AWARENESS TO CONTROL THE OVERLAY IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP2 / 210120-2 U.S. Patent Application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP3 / 210120-3 • U.S. Patent Application entitled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP4 / 210120-4 • U.S. Patent Application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Agent Reference Number END9352USNP5 / 210120-5 • U.S. Patent Application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; Agent Reference Number END9352USNP7 / 210120-7 U.S. Patent Application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; Agent Reference Number END9352USNP8 / 210120-8 U.S. Patent Application entitled "Cooperative Overlays of Interacting Instruments Which Resurface in Both Overlays Being Effected"; Agent Reference Number END9352USNP9 / 210120-9 U.S. Patent Application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Agent Reference Number END9352USNP10 / 210120-10 U.S. Patent Application entitled "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN"; Agent Reference Number END9352USNP11 / 210120-11 U.S. Patent Application entitled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Agent Reference Number END9352USNP12 / 210120-12 U.S. Patent Application entitled "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION"; Agent Reference Number END9352USNP13 / 210120-13 U.S. Patent Application entitled "Cooperation Among Multiple Display Systems to Provide a Healthcare User Customized Information"; Agent Reference Number END9352USNP14 / 210120-14 • U.S. patent application entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP15 / 210120-15, • U.S. patent application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP16 / 210120-16, and • U.S. patent application entitled "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS"; Attorney Docket No. END9352USNP17 / 210120-17.

[0010] The assignee of the present application owns the following U.S. patent applications, the entire disclosures of each of which are incorporated herein by reference. • U.S. Patent Application No. 16 / 209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS" (now U.S. Patent Application Publication No. 2019 / 0200981-A1), • U.S. Patent Application No. 16 / 209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (now U.S. Patent Application Publication No. 2019 / 0201046-A1).

[0011] Before describing various aspects of surgical devices and generators in detail, it should be noted that the illustrative embodiments are not limited to the details of the construction and arrangement of parts illustrated in the accompanying drawings and description in application or use. The illustrated embodiments may be implemented in or incorporated into other aspects, variations, and modifications, and may be practiced or carried out in various ways. Furthermore, unless otherwise specified, the terms and expressions used herein are selected for the purpose of describing the illustrated embodiments for the convenience of the reader, and are not intended to limit the same. Furthermore, it should be understood that any one or more of the aspects, expressions of aspects, and / or embodiments described below may be combined with any one or more of the other aspects, expressions of aspects, and / or embodiments described below.

[0012] Various aspects are directed to on-screen displays for surgical systems for a variety of energy-based and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio frequency (RF)-based monopolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers, among others. Surgical stapler devices include surgical staplers combined with electrosurgical devices and / or ultrasonic devices. Aspects of ultrasonic surgical devices may, for example, be configured to transect and / or coagulate tissue during a surgical procedure. Aspects of electrosurgical devices may, for example, be configured to transect, coagulate, seal, weld and / or desiccate tissue during a surgical procedure. Aspects of surgical stapler devices may be configured to transect and staple tissue during a surgical procedure, and in some aspects, a surgical stapler device may be configured to deliver RF energy to tissue during a surgical procedure. Electrosurgical devices are configured to deliver therapeutic and / or non-therapeutic RF energy to tissue. Elements of surgical staplers, electrosurgical devices, and ultrasonic devices may be used in combination in a single surgical instrument.

[0013] In various embodiments, the Disclosure provides the OR team with on-screen displays of real-time information during surgical procedures. According to various embodiments of the Disclosure, many novel and unique on-screen displays are provided for displaying various visual information feedback to the OR team on screen. According to the Disclosure, the visual information may include one or more of various visual media, with or without sound. Generally, the visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. The visual information may be communicated on any number of display options, such as, for example, a primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, etc.

[0014] In various embodiments, this disclosure provides a list of many potential options for communicating visual information to an OR team in real time without overwhelming the OR team with too much visual information. For example, in various embodiments, this disclosure provides on-screen displays of visual information that enable a surgeon, or other member of the OR team, to selectively activate on-screen displays, such as icons surrounding screen options, to manage the rich visual information. One or a combination of factors may be used to determine the active display, and these may include, among other things, the energy-based (e.g., electrosurgery, ultrasound) or machine-based (e.g., stapler) surgical device in use, the estimated risk associated with a given display, the surgeon's level of experience, and the surgeon's choice. In other embodiments, the visual information may include rich data overlaid or superimposed on the surgical field to manage the visual information. In various embodiments described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. Visual information data thus communicated can provide additional useful visual information to the OR team in a more concise and understandable way, in contrast to static icons.

[0015] In various embodiments, the Disclosure provides techniques for selectively activating on-screen displays, such as icons surrounding a screen, to manage visual information during a surgical procedure. In other embodiments, the Disclosure provides techniques for determining an active display using one or a combination of factors. In various embodiments, the techniques provided by the Disclosure may include, among other things, selecting an energy-based or machine-based surgical device to be used as the active display, estimating the risks associated with a given display, and utilizing the experience level of the surgeon or OR team making the selection.

[0016] In other embodiments, the techniques described herein may include overlaying or superimposing rich data onto the surgical field for the purpose of managing visual information. Several display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Graphical overlays may take the form of transparent graphics, translucent graphics, or opaque graphics, or combinations of transparent, translucent, and opaque elements or effects. Furthermore, overlays may be positioned on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value. Graphical overlays are rendered on an active display monitor to quickly and efficiently communicate critical information to the OR team.

[0017] In other embodiments, the technology provided by the Disclosure may include superimposing images that require video analysis and tracking in order to appropriately overlay visual information data. In other embodiments, the technology provided by the Disclosure may include communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easily understandable manner. In other embodiments, the visual overlay may be used in combination with auditory and / or somatosensory overlays, e.g., thermal, chemical, and mechanical devices, and combinations thereof.

[0018] The following description generally pertains to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information onto the real-world images of the surgical field, instruments, and / or other objects appearing in the surgical field. The images may be streamed in real time or they may be still images. Augmented reality is a technique for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid on a real environment. The real environment may include the surgical field. Virtual objects overlaid on a real environment may be represented at fixed or set positions relative to one or more aspects of the real environment. In non-limiting examples, if a real-world object moves out of the field of view of the real environment, the virtual object fixed to the real-world object also moves out of the field of view of augmented reality.

[0019] Some of the display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value.

[0020] As described herein, AR is an extended version of the real physical world achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear real, making the user feel immersed in them. This environment is perceived through a device known as a virtual reality headset or helmet. While both mixed reality (MR) and AR are considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. AR often adds digital elements to a live view by using a camera, while an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

[0021] In an AR environment, one or more computer-generated virtual objects may be displayed alongside one or more real-world (i.e., so-called "real-world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlay virtual objects. Such virtual objects can provide supplementary information about the environment or, in general, enhance the user's perception and engagement with the environment. Conversely, real-time images or videos of the surrounding environment can, in addition or alternatively, enhance the user's engagement with the virtual objects displayed on the display.

[0022] Apparatus, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during surgical procedures. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and / or cameras that provide images during incisional surgical procedures. Images may be streamed in real time or still images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing images of a real-world surgical environment by overlaying representations of virtual objects or data and / or real objects onto the real-world surgical environment. The augmented reality experience may be viewed on a display and / or AR device that allows the user to view virtual objects overlaid on the real-world surgical environment. The display may be located in the operating room or located away from the operating room. The AR device is worn on the head of a surgeon or other operating room personnel and typically includes two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through the two transparent or translucent display lenses so that aspects of the real environment are visible, while projecting light to make virtual objects visible to the user of the AR device.

[0023] Two or more displays and AR devices may be used in conjunction with a first display or AR device that controls one or more additional displays or AR devices in a system having defined roles. For example, when activating a display or AR device, the user may select a role (e.g., a surgeon, surgical assistant, nurse, etc. during a surgical procedure), and the display or AR device may display information related to that role. For example, a surgical assistant may have the display show virtual representations of instruments that the surgeon needs to use for the next step in the surgical procedure. The surgeon's focus on the current step may differ from the information displayed by the surgical assistant.

[0024] While many known on-screen displays and alerts exist, this disclosure provides many novel and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment may be provided to the operating room (OR) team, including, but not limited to, personnel within the OR, such as the surgical surgeon, surgical assistants, scrub wearers, anesthesiologists, and circulating nurses. The virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical staplers, tablets, augmented reality glasses, and other devices.

[0025] Figure 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 connected to remote storage 5. Each surgical system 2 comprises at least one surgical hub 6 that communicates with the cloud 4. For example, a surgical system 2 may comprise a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12, each configured to communicate with each other and / or with the hub 6. In some embodiments, a surgical system 2 may comprise M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12, where M, N, O, and P are integers of 1 or more. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during surgical procedures, as described herein.

[0026] Figure 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. A robotic system 10 is used as part of the surgical system 2 in the surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient-side cart 20 (surgical robot), and a surgical robot hub 22. The patient-side cart 20 allows the surgeon to operate at least one detachably connected surgical tool 17 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. Images of the surgical site during the minimally invasive procedure (e.g., still or live images streamed in real time) can be acquired by a medical imaging device 24. The patient-side cart 20 can operate the imaging device 24 to orient it. Images of the incision surgical procedure can be acquired by a medical imaging device 96. The robot hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18, or on an AR device 66 worn by the surgeon or another person in the surgical room 16.

[0027] The optical components of the imaging device 24, 96, or AR device 66 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from tissues and instruments in the surgical field.

[0028] In various embodiments, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes. In various embodiments, the imaging device 96 is configured for use in incisional (invasive) surgical procedures.

[0029] In various embodiments, the visualization system 8 comprises one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more displays. In one embodiment, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one embodiment, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range in the electromagnetic spectrum. Wavelengths are separated by filters or by instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring allows the surgical field to be repositioned after the surgical task for performing tests on the treated tissue is completed.

[0030] Figure 2 shows a primary display 19 positioned in the sterile field for the operator on the operating table 14 to see. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing opposite directions from each other. The visualization system 8, guided by the hub 6, is configured to utilize displays 7, 9, and 19 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by imaging devices 24 and 96 through the non-sterile displays 7, 9, or AR device 66, while maintaining live video of the surgical site on the primary display 19 or AR device 66. The non-sterile displays 7 and 9 can, for example, enable non-sterile operators to perform diagnostic steps related to the surgical procedure.

[0031] Figure 3 shows a hub 6 that communicates with a visualization system 8, a robotic system 10, and handheld intelligent surgical instruments 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke extraction module 26 and / or a suction / irrigation module 28. In various embodiments, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in Figure 10). Modular light sources can be adapted for use with various imaging devices. In various examples, multiple imaging devices can be positioned at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or incisional surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide the optimal view. In various embodiments, the imaging module 38 can be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during surgical procedures as described herein.

[0032] Figure 4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located in one or more operating rooms / surgery sites of a medical facility to a cloud-based system. The cloud 54 may include a remote server 63 (Figure 5) connected to a storage device 55. The modular communication hub 53 includes a network hub 57 and / or a network switch 59 that communicate with a network router 61. The modular communication hub 53 is connected to a local computer system 60 for data processing. The modular devices 1a-1n in the operating room may be connected to the modular communication hub 53. The network hub 57 and / or the network switch 59 are connected to the network router 61 so that the devices 1a-1n can connect to the cloud 54 or the local computer system 60. Data associated with the devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 via a wired channel or a wireless channel. The surgical data network environment 51 may be employed, as described herein, to provide an augmented reality interactive experience during a surgical procedure, in particular to provide augmented images of the surgical field to one or more remote displays 58.

[0033] Figure 5 shows a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 which may include a remote server 63. In one embodiment, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to a plurality of surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 connected to a computer system 60.

[0034] Returning to Figure 5, the modular control tower 23 is connected to an imaging module 38 connected to an endoscope 98, a generator module 27 connected to an energy device 99, a fume exhaust module 76, a suction / irrigation module 78, a communication module 13, a processor module 15, a storage array 16, and optionally smart devices / instruments 21 and sensor modules 29 connected to a display 39. Surgical site devices are connected to cloud computing resources such as a server 63, data storage 55, and a display 58 via the modular control tower 23. The robot hub 72 may also be connected to the modular control tower 23, as well as the server 63, data storage 55, and display 58. In particular, the devices / instruments 21 and the visualization system 58 may be connected to the modular control tower 23 via wired or wireless communication standards or protocols as described herein. The modular control tower 23 may be connected to a hub display 65 (e.g., a monitor, screen) to display received augmented images, including overlaid virtual objects on the real surgical world, received from the imaging module 38, device / instrument display 39, and / or other visualization systems 58. The hub display 65 may also display data received from devices connected to the modular control tower 23, along with the images and overlay images.

[0035] Figure 6 shows a surgical hub 56 including multiple modules connected to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, such as a network connectivity device, and a computer system 60 for local processing, visualization, and imaging of augmented surgical information. The modular communication hub 53 is connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53, and data associated with the modules may be transferred to the computer system 60, cloud computing resources, or both. Each of the network hubs / switches 57 / 59 within the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs / switches 57, 59 are connected to the processor 31 to provide communication connectivity to cloud computing resources and local displays 67. Communication to the cloud 54 can be done via either a wired communication channel or a wireless communication channel.

[0036] The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is connected via a system bus to a communication module 41, storage 45, memory 46, non-volatile memory 47, and an input / output interface 48. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using various available bus architectures.

[0037] The processor 31 may include an augmented reality modeler (e.g., as shown in Figure 10) and may be implemented as a single-core or multi-core processor, such as one known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0038] System memory includes volatile and non-volatile memory. The Basic Input / Output System (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory may include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0039] The computer system 60 also includes removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0040] In various embodiments, the computer system 60 in Figure 6, the imaging module 38 in Figures 4 to 6, and / or the visualization system 58, and / or the processor module 15 may include an image processor, an image processing engine, an image processing unit (GPU), a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0041] Figure 7 shows an augmented reality system 263 that includes an intermediate signal coupler 64 located in the communication path between the imaging module 38 and the surgical hub display 67. The signal coupler 64 combines audio and / or image data received from the imaging module 38 and / or the AR device 66. The surgical hub 56 receives the combined data from the coupler 64, overlays the provided data onto the display 67, and displays the overlaid data. The imaging device 68 may be a digital video camera, and the audio device 69 may be a microphone. The signal coupler 64 may include a wireless head-up display adapter for coupling to the AR device 66 located in the communication path of the display 67 to a console that enables the surgical hub 56 to overlay data onto the display 67.

[0042] Figure 8 shows an augmented reality (AR) system including an intermediate signal coupler positioned in the communication path between the imaging module and the surgical hub display. Figure 8 shows an AR device 66 worn by the surgeon 73 to communicate data to the surgical hub 56. Peripheral information of the AR device 66 does not include active images. Rather, peripheral information includes only signals that do not have the same requirements for device settings or refresh rate. The interaction may extend the surgeon 73's information based on links with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures and, for example, ask whether an instrument is touching a nerve, blood vessel, or adhesion. The AR device 66 may include processing in the surgical hub 56 used to provide preoperative scan data, optical views, tissue examination characteristics acquired throughout the procedure, and / or answers. The surgeon 73 may write notes on the AR device 66 so that they are stored in the hub storage 45 along with patient data for later use in reporting or follow-up.

[0043] The AR device 66, worn by the surgeon 73, links to the surgical hub 56 using auditory and visual information to avoid the need for overlays, and allows customization of information displayed around the periphery of the field of view. The AR device 66 provides signals from devices (e.g., instruments) and answers queries regarding location information linked with video to identify device settings or quadrants or locations. The AR device 66 has voice control and voice feedback from the AR device 66. The AR device 66 can interact with other systems in the operating room and can have available feedback and interaction wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture start commands and queries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities, including voice, visual, or haptic touch.

[0044] Figure 9 shows a surgeon 73 and a patient 74 wearing AR devices 66, and may include a camera 96 ​​in the operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects overlaid on a real-time image of the surgical field to the surgeon 73 through an augmented reality display 89 or through a hub-connected display 67. The real-time image may include parts of surgical instruments 77. The virtual objects may not be visible to others in the operating room 75 (e.g., surgical assistants or nurses), but they may also wear AR devices 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects, or may be able to see the virtual objects in augmented reality shared with the surgeon 73, or may be able to see modified versions of the virtual objects (e.g., according to customization specific to the surgeon 73), or may see different virtual objects.

[0045] Virtual objects and / or data may be configured to appear on a portion of the surgical instrument 77 or within the surgical field captured by the imaging module 38, the imaging device 68 during minimally invasive surgical procedures, and / or the camera 96 ​​during incisional surgical procedures. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides live images of the surgical area during minimally invasive surgical procedures. The AR system may present virtual objects fixed to real objects regardless of the viewpoint of one or more viewers of the AR system (e.g., the surgeon 73). For example, virtual objects may be visible to viewers of the AR system inside the operating room 75, but not visible to viewers of the AR system outside the operating room 75. Virtual objects may be displayed to viewers outside the operating room 75 when a viewer enters the operating room 75. Augmented images may be displayed on the surgical hub display 67 or the augmented reality display 89.

[0046] The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each user's eye). The screens may allow light to pass through them so that aspects of the real environment are visible while virtual objects are being displayed. Virtual objects may become visible to the surgeon 73 by projecting light. Virtual objects may appear to have some degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

[0047] An AR system may be visible to one or more viewers and may include differences between views available to one or more viewers, while maintaining some common aspects between views. For example, a heads-up display may change between two views, but virtual objects and / or data may be fixed to real objects or areas in both views. Aspects such as the color, lighting, or other changes of an object may occur between views without changing the fixed position of at least one virtual object.

[0048] Users can view virtual objects and / or data presented within the AR system as opaque or with a certain level of transparency. For example, a user can interact with a virtual object by moving it from a first position to a second position. For instance, a user may move an object with their own hand. This may be done virtually in the AR system by determining that the hand has moved to a position adjacent to or adjacent to the object (using one or more cameras, which may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controlled to move) and moving the object accordingly. The virtual form may include a virtual representation of a real-world object, or it may include visual effects such as lighting effects. The AR system may include rules to govern the behavior of the virtual object, such as exposing the virtual object to gravity or friction, or it may include other predefined rules that negate real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device 66 may include a camera 79 (which should not be confused with a separate camera 96). The AR device camera 79 or camera 96 ​​may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. The AR device 66 may project virtual items onto a representation of the real environment that the user can see.

[0049] The AR device 66 may be used, for example, in an operating room 75 during a surgical procedure performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects, such as virtual objects during the surgical procedure, to extend the surgeon's vision. The surgeon 73 may view the virtual objects using the AR device 66, a remote controller for the AR device 66, or interact with the virtual objects by using his hands to “interact” with the virtual objects or gestures recognized by the camera 79 of the AR device 66, for example. The virtual objects can extend surgical tools, such as surgical instruments 77. For example, the virtual object may appear to be connected to the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66), or to remain at a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide the surgical instrument 77 and may appear to be fixed to the patient 74. In certain examples, the virtual object may react to the movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

[0050] The augmented reality display system imaging device 38 captures real images of the surgical area during the surgical procedure. The augmented reality displays 89 and 67 present an overlay of the operating modes of the surgical instrument 77 onto the real images of the surgical area. The surgical instrument 77 includes a communication circuit 231 for communicating operating mode and functional data from the surgical instrument 77 to the AR device 66 via a communication circuit 233 on the AR device 66. The surgical instrument 77 and the AR device 66 are shown in RF wireless communication between circuits 231 and 233, as indicated by arrows B and C, but other communication technologies (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operating modes of the surgical instrument 77 that are actively visualized. The overlay combines the modes of tissue interaction in the surgical area with functional data from the surgical instrument 77. The processor portion of the AR device 66 is configured to receive operating mode and functional data from the surgical instrument 77, determine overlays related to the operation of the surgical instrument 77, and combine the tissue characteristics within the surgical area with the functional data from the surgical instrument 77. The augmented images display alerts regarding device performance considerations, non-conformity use, and incomplete capture. Non-conformity use includes out-of-range tissue conditions and tissue improperly balanced within the jaws of the end effector. Additional augmented images provide indications of incidental events, including indications of tissue tension and foreign body detection. Other augmented images display device status overlays and instrument indications.

[0051] Figure 10 shows a system 83 for augmenting images of the surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes one aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, memory 87, an AR display 89, and a camera 79. The AR device 66 may also include a sensor 90, a speaker 91, and / or a haptic controller 92. The database 93 may include image storage 94 or preoperative planning storage 95.

[0052] The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive images of instruments in the surgical field from a camera 79 or sensor 90, etc., and create an augmented reality environment that fits within the displayed image of the surgical field. In another example, physical objects and / or data may be overlaid on the surgical field and / or surgical instrument images, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of virtual objects and / or data within the augmented reality environment. For example, the augmented reality modeler 86 may use or detect instruments at the patient's surgical site and present virtual objects and / or data on the surgical instruments, and / or images of the surgical site in the surgical field captured by the camera 79. The AR display 89 may display the AR environment overlaid on the real environment. The display 89 can use the AR device 66, which is located in a fixed position, etc., within the AR environment, to show virtual objects and / or data.

[0053] The AR device 66 may include sensors 90 such as infrared sensors. The camera 79 or sensor 90 may be used to detect movements such as gestures by a surgeon or other user, which may be interpreted by the processor 85 as attempted or intended interactions by the user with a virtual target. The processor 85 can identify objects in the real environment by processing information received using the camera 79, for example. In other embodiments, sensor 90 may be a tactile sensor, audible sensor, chemical sensor, or thermal sensor to generate corresponding signals that can be combined with various data feeds to create an augmented environment. Sensor 90 may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye-tracking sensors, and voice command recognition capabilities.

[0054] The AR display 89 may, for example, during a surgical procedure, allow the surgical field to be viewed through the AR display 89, while presenting virtual features within the surgical field that correspond to physical features hidden by the patient's anatomical features. The virtual features may have a virtual position or orientation that corresponds to a first physical position or orientation of the physical features. In one example, the virtual position or orientation of the virtual features may include an offset from the first physical position or orientation of the physical features. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical features, and so on.

[0055] In one example, the AR device 66 may be an individual AR device. In one embodiment, the AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft in Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometer, gyroscope, magnetometer), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition capabilities. It provides a high-resolution, improved field of view by using mirrors to orient waveguides in front of the wearer's eyes. The image can be magnified by changing the angle of the mirrors. It also provides eye tracking to recognize the user and adjust the lens width for a particular user.

[0056] In another example, AR device 66 could be the Snapchat Spectacles 3 AR device. This AR device offers the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras for capturing 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to construct a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synchronized to an external display device.

[0057] In yet another example, AR device 66 could be Google's Glass 2 AR device. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

[0058] In another example, AR device 66 could be Amazon's Echo Frames AR device. This AR device does not have a camera / display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

[0059] In yet another example, AR device 66 could be the Focals AR device by North (Google). This AR device provides a notification pusher / smartwatch analog, inertial measurement, screen overlays for information (weather, calendar, messages), and voice control (Alexa) integration. This AR device also provides basic head-up display functionality.

[0060] In another example, AR device 66 could be an Nreal AR device. This AR device includes spatial sound, two ambient cameras, a photographic camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and proximity sensor functions. Nebula projects application information onto the lens.

[0061] In various other examples, the AR device 66 may be any one of the following commercially available AR devices, namely Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, or EyeTap, which create light collinear with the ambient light directly onto the retina. A beam splitter makes the same visible light available to a computer, for example, to process and overlay information. The AR visualization system may include a HUD, contact lenses, glasses, virtual reality (VR) headset, virtual retinal display, intraoperative display, and / or smart contact lenses (bionic lenses).

[0062] The multi-user interface for the AR device 66 includes a virtual retinal display such as a raster display that draws directly onto the retina rather than on a screen in front of the eyes, a smart TV, a smartphone, and / or a spatial display such as the Sony Spatial Display System.

[0063] Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, the Apple Polycam app and Ubiquity 6 (Mirrorworld using the Display.land app), which allow users to scan and acquire 3D images of the real world (to create 3D models). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, and Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, and Magic Leap.

[0064] Situational awareness is the ability of several embodiments of a surgical system to determine or infer information related to a surgical procedure from data received from a database and / or instruments. This information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. Based on contextual information relating to a surgical procedure, the surgical system can be improved, for example, by controlling modular devices connected to it (e.g., robotic arms and / or robotic surgical tools) and providing contextualized information or suggestions to the surgeon during the course of the surgical procedure.

[0065] Figure 11 shows a timeline of a situation-aware surgical procedure. Figure 11 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each step of the surgical procedure. Timeline 5200 shows typical steps that nurses, surgeons, and other healthcare workers might take during a lung segmentectomy procedure, which begins with setting up the operating room and ends with transferring the patient to the postoperative recovery room. Throughout the course of the surgical procedure, the situation-aware surgical hub 5104 receives data from data source 5126, including data generated each time healthcare workers use the modular device 5102 paired with the surgical hub 5104. By receiving this data from the paired modular device 5102 and other data sources 5126, the surgical hub 5104 can continuously derive estimations (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub 5104 can, for example, record data relating to a procedure to generate a report, verify the steps being taken by a healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as described above.

[0066] In the first step 5202, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery.

[0067] In the second 5204, staff scan incoming medical supplies for a procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies used in various types of procedures to confirm that the mixture of supplies corresponds to a thoracic procedure. Furthermore, the surgical hub 5104 can also determine that the procedure is not a wedge resection (because the incoming supplies either do not contain specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection).

[0068] In the third 5206, a healthcare worker scans the patient band via a scanner 5128 that is communicably connected to a surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

[0069] In the fourth part of 5208, a medical professional turns on the assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they include a fume exhauster, an air inlet, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104 located within a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 determines that the surgical procedure is a VATS surgery based on this particular combination of paired modular devices 5102. Based on the combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team. Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing.

[0070] In step 5210, the staff attaches the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 confirms that the patient is in the operating room.

[0071] In step 6, 5212, medical personnel induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from modular devices 5102 and / or patient monitoring devices 5124, including, for example, EKG data, blood pressure data, ventilator data or a combination thereof. Once step 6, 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion commences.

[0072] In section 7 of 5214, the lung of the patient being operated on collapses (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has begun and determine that collapsing the lung is the first surgical step in this particular procedure.

[0073] In step 8, 5216, a medical imaging device 5108 (e.g., a scope) is inserted, and video footage from the medical imaging device is initiated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or real-time live-streaming video) through its connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resection has not been taken into consideration by the surgical hub 5104 based on the data received in step 2 of the procedure, 5204). Using data from the medical imaging device 124 (Figure 2), contextual information regarding the type of procedure being performed can be determined in various ways, for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number of medical imaging devices being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used.

[0074] For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. The situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0075] In step 9, 5218, the surgical team initiates the incision step of the procedure. The surgical hub 5104 receives data from the RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the procedure steps described above) corresponds to the incision step.

[0076] In the tenth step 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 receives data from the surgical stapling and cutting instruments indicating that instruments are being fired, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps in the read-out process.

[0077] In the eleventh step 5222, the segmental resection portion of the procedure is performed. The surgical hub 5104 estimates that the surgeon has transversely incised parenchymal tissue, based on data from surgical instruments, including data from a staple cartridge. Cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Cartridge data may indicate the type of tissue being stapled and / or transversely incised, for different types of staples used for different types of tissue. The type of staples fired is used for parenchymal tissue or other similar tissue types, and the surgical hub 5104 can estimate that a segmental resection procedure has been performed.

[0078] Next, in the twelfth step 5224, the nodule incision step is performed. Based on the data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby enabling the surgical hub 5104 to make this inference. Note that the surgeon will periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once the twelfth step 5224 is completed, the incision is closed and the postoperative portion of the procedure begins.

[0079] In the 13th step 5226, the patient is de-anesthetized. The surgical hub 5104 can estimate that the patient is waking up from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase).

[0080] Finally, in the 14th step 5228, the healthcare worker removes the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 can infer that the patient has been transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. Based on the data received from various data sources 5126 that are communicably connected to the surgical hub 5104, the surgical hub 5104 can determine or infer when each step of a given surgical procedure is occurring.

[0081] As shown in the first step 5202 of the timeline 5200 shown in Figure 11, in addition to using patient data from the EMR database(s) to estimate the type of surgical procedure being performed, the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0082] Surgical displays (e.g., displays 7, 9, 19, 35, 62, 65, 66, 67, and 89) play a vital role in the operating room by providing useful information to clinicians (e.g., surgeons, surgical staff), which can be used, among other things, to assess the progress of surgical procedures, determine subsequent steps for performing surgical procedures, and monitor the patient's vital signs. The displays need to be large enough to see the information provided, but not excessively large, so as not to hinder workflow or movement in a crowded operating room.

[0083] For example, an imaging device, such as one of the many imaging devices described elsewhere in this specification, is used to capture a live stream of the surgical field during a surgical procedure. A display shows this live stream captured by the imaging device so that the clinician can view the surgical field during the surgical procedure.

[0084] During a surgical procedure, information related to or associated with the procedure may be overlaid on a live stream on a display. For example, an electrocardiogram (EKG) may monitor the patient's heart rate during the procedure, and the monitored heart rate may be overlaid on the live stream to ensure that the clinician can assure the patient that the patient is stable.

[0085] Various other sensors, detectors, and modules can monitor other parameters throughout the surgical procedure, and information associated with these parameters can also be overlaid on the display. However, some overlaid information may be more important than others. For example, when a clinician is manipulating tissue using the end effector of a surgical instrument, information about the amount of force being applied to the tissue using the end effector is relevant to monitoring to ensure that the tissue is not unintentionally damaged.

[0086] However, due to the sheer volume of information overlaid on the display, more important information, such as the forces being applied to tissue, may be overlooked or missed by clinicians. This abundance of competing information can overwhelm surgeons with information that may be detrimental to their ability to perform surgical procedures properly and may prove costly for the patient. Therefore, the amount of data / information overlaid on the display needs to be prioritized, controlled, and / or limited.

[0087] Figure 12 shows a surgical visualization system 6000 according to at least one aspect of the present disclosure. Various components of the surgical visualization system 6000 are in many respects similar to components of other systems described elsewhere in the present disclosure and are therefore not repeated herein at the same level of detail for the sake of brevity. The surgical visualization system 6000 includes a control module 6001 configured to perform various techniques described herein by using one or more processors or processing circuits, such as a processor 85. In some embodiments, the system 6000 may include, be used with, or communicate with, an augmented reality device 85, for example. The system 6000 may further include a storage medium, such as a memory 6003, an imaging device 6004, such as a camera 88, and a display 6005. The system 6000 may further include one or more speakers 91, a tactile controller 92, and / or sensors 90 (see Figure 10). The display 6005 may include, for example, an AR display 89, a VR display, a projector, a head-up display, a screen, and / or any other suitable device for displaying visual content.

[0088] In some embodiments, system 6000 is incorporated, for example, into a computer-implemented interactive surgical system 50. In some embodiments, system 6000 operably communicates with one or more hubs, systems, networks, servers, and / or databases that can deliver surgical data to system 6000. For example, system 6000 may operably communicate with a cloud 54, which may include a remote server 63, a robot hub 72, a surgical hub 56, devices / instruments 21, and / or a modular control tower 23, via wired or wireless communication standards or protocols as described herein. In some embodiments, system 6000 includes a context-aware module 6006 similar to that described in relation to the surgical hub 5104. The context-aware module 6006 may be trained to extrapolate contextual information about surgical procedures based on a large amount of perioperative data received through sensor inputs and / or user inputs.

[0089] In consideration of the aforementioned problems associated with competition for the amount of information overlaid on a display, the Disclosure provides a system, such as System 6000, that can monitor, sense, and / or detect the occurrence of trigger events occurring before, during, or after a surgical procedure in order to control the information overlaid on a display. In one embodiment, the trigger event may be an event detected by the System via any number of sensors, systems, or modules, as described elsewhere herein, that can initiate a change in the information overlaid on the display. In various embodiments, the detection of a trigger event may cause information to be added to the display, removed from the display, or, for example, moved to a different location on the display, or adjusted in size, as described in more detail elsewhere herein.

[0090] In one embodiment, the system can detect recognition-based triggers via a surgical visualization system such as visualization system 8 and update the overlaid information on the display accordingly. In various embodiments, the visualization system 8 may be analogous to the visualization systems described in U.S. Patent No. 11,000,270, U.S. Patent Application Publication No. 2020 / 0015900, U.S. Patent Application Publication No. 2020 / 0015899, U.S. Patent Application Publication No. 11,259,793, U.S. Patent Application Publication No. 2020 / 0015924, U.S. Patent Application Publication No. 2020 / 0015898, U.S. Patent Application Publication No. 2020 / 0015906, U.S. Patent Application Publication No. 2020 / 0015907, U.S. Patent No. 10,925,598, U.S. Patent Application Publication No. 2020 / 0015914, and U.S. Patent Application Publication No. 2020 / 0015902, which are incorporated herein by reference in their entirety.

[0091] In one embodiment, the recognition-based trigger may be, for example, an object having a predetermined and / or identifiable size, shape, pattern, color, arrangement, or any other identifiable feature unique to the object (e.g., surgical instruments, surgical implants, surgical structures, organs, tissues). In various embodiments, the system may include a memory such as memory 6003, which stores therein images and / or data associated with the object, such as parameters, for comparison with the object captured by an imaging device such as imaging device 6004 during a surgical procedure. In one embodiment, the memory may store two-dimensional images of the object therein, such as a top view, bottom view, side view, isometric view, or any other suitable two-dimensional view of the object. In one embodiment, the memory may store three-dimensional models of the object therein, such as a CAD model, so that any number of image views are available to the system for comparison. In one embodiment, the three-dimensional model may be generated using a visualization system 8 with preoperative imaging techniques such as a CT scan or MRI scan.

[0092] In one exemplary embodiment, the system can identify objects in a live stream via an imaging device. To determine whether an object is a known object, the system can compare an image of an object that can be identified by the system and its parameters (color, dimensions, etc.) with images and parameters stored in memory. If there is a match, or at least a substantial match, the system can overlay information associated with the object identified in the live stream onto a display.

[0093] In one exemplary embodiment, an imaging device can capture natural surface features, such as gastric corner notches, within a live stream. The system can transmit a visual representation of the live stream to a display so that the natural surface features can be viewed by surgical staff. The system can further compare the image and determined parameters of the natural surface features with images and parameters stored in memory to determine whether the natural surface features are known. In the case of positive identification of a natural surface feature, the system can overlay information associated with the natural surface feature on the display. In one embodiment, the information associated with the natural surface feature may be stored in memory. In one embodiment, the overlaid information may be overlaid on the natural surface feature on the display. In one embodiment, the overlaid information may be overlaid close to the natural surface feature on the display so that the overlaid information is easily visible but does not obstruct the view of the natural surface feature on the display. In one embodiment, the overlaid information may be overlaid at a predetermined location on the display designated for positive identification in the live stream, such as the corner of the display.

[0094] In one embodiment, as described above, the objects in the livestream may be natural surface features. In one embodiment, the objects in the livestream may be surface features of surgical instruments such as surgical staple cartridges. In one embodiment, the objects in the livestream may be markers such as barcodes, emblems, or patterns. In one embodiment, the objects in the livestream may be any number of objects that the system can compare with images and parameters of objects stored in memory.

[0095] In one embodiment, the system may overlay information on the display based on partial identification of objects in a live stream. In another embodiment, the system may identify objects in a live stream that meet a threshold tolerance and overlay information on the display if the threshold tolerance is reached or exceeded. In another embodiment, the threshold tolerance may be predefined, stored in memory, defined by the user, based on industry standards, or a combination thereof. If the threshold tolerance is not reached, the system may not overlay information on the display.

[0096] In one exemplary embodiment, the system can identify a portion of a staple cartridge in a live stream. In one embodiment, the staple cartridge may be obstructed or partially out of frame in the live stream so that only a portion of the staple cartridge is visible. The system can compare the visible portion of the staple cartridge with images and parameters of the staple cartridge stored in memory. In one embodiment, the parameters of the staple cartridge may be visible / identifiable dimensions of the cartridge, such as the color of the cartridge, the distance between staple cavities or the length of the elongated slots traversed by the cutting knife, the number of staple cavities, or any other identifiable parameters associated with the staple cartridge. As will be described in more detail below, if the system determines that a portion of the staple cartridge reaches or exceeds a threshold tolerance when compared with surgical staple cartridges stored in memory, the system can overlay information on the display based on the determination.

[0097] In some embodiments, the threshold tolerance limit can be defined as the percentage of images or their parameters stored in memory that are identified within the live stream. In one exemplary embodiment, the system can identify a portion of the staple cartridges in the live stream. The system can analyze the images and determine that 75% of the staple cartridges stored in memory are identified on the objects from the live stream. In one embodiment, the system may have a threshold tolerance limit of 50% that is exceeded, for example, by a comparison between the objects in the live stream and the images stored in memory. Thus, information associated with the staple cartridges can be overlaid on the display. In various embodiments, the tolerance limit can be stored in memory, can be defined by the user, can vary from user to user, can be based on standard industry practices, or a combination thereof.

[0098] In some embodiments, the threshold tolerance may be defined as a threshold number of identified parameters based on a comparison between objects identified in the live stream and objects stored in memory. In one exemplary embodiment, the system can identify a portion of a staple cartridge in the live stream. The system can identify various parameters of the staple cartridge, such as color, spacing between staple cavities, known marks on the staple cavities, or any other identifiable features of the staple cartridge. The system can identify these parameters and compare them to parameters stored in memory, such as parameters stored in a lookup table. In one embodiment, the threshold tolerance may be set to three matches between objects identified in the live stream and objects stored in memory. If the system determines that the threshold tolerance has been reached or exceeded (for example, identifying the color of a staple cartridge, identifying the spacing between staple cavities, seeing a known emblem on it, etc.), the system can overlay information on the display according to the matches. In various embodiments, the threshold tolerance may be a combination of the proportion of objects identified in the live stream and the number of parameters of the identified objects. In one exemplary embodiment, the threshold tolerance may be such that 50% of the objects in the live stream match objects stored in memory, and three parameters match objects stored in memory.

[0099] In one embodiment, the system may overlay a confidence level associated with an identified match. As described herein, the system may identify a partial match in the live stream and overlay information when a threshold tolerance is reached or exceeded. In the case of a partial match, the system may overlay a confidence level or percentage with the overlaid information. In one exemplary embodiment, a staple cartridge stored in memory may have eight parameters associated with it, but the threshold tolerance is set to only three matches. If the system identifies three positive matches of the eight parameters in the staple cartridge in the live stream, the system may overlay information about the staple cartridge on the live stream. In addition, the system may overlay a note indicating that the overlay is based on three of the eight identified parameters, i.e., not an exact match. By overlaying confidence levels, surgical personnel viewing the display can utilize their own judgment regarding whether they agree with the decision. In various embodiments, the system may include a user interface that allows surgical staff to accept or reject overlaid information, thereby giving staff the ability to remove the overlaid information if they do not agree to the assessment or do not need the overlaid information.

[0100] In various embodiments, the system can overlay information onto a live stream according to identified objects on the live stream. In one embodiment, the system can overlay markers that identify different regions or features of an object based on positive identification. In an exemplary embodiment, if the system identifies an object as a stomach, the system can overlay markers that point to, for example, the greater curvature, lesser curvature, and angular notch. In one embodiment, the system can overlay segmented overlays onto the object that identify different regions of the object. In an exemplary embodiment, if the system identifies the stomach, it can overlay segmented overlays that identify, for example, the fundus, body, pyloric sac, pyloric canal, and inside the duodenum.

[0101] In one embodiment, the system can overlay directional information onto a live stream based on positive identification. In an exemplary embodiment, if the system identifies a corneal notch, the system can overlay a directional arrow to help the surgeon locate other areas of the stomach, such as the greater curvature, or other organs within the patient, such as the intestines. In one embodiment, the directional arrow can be based on both the identified object and the orientation or angle in which the object was identified. In some embodiments, the directional arrow can be based on a determined step of the surgical procedure. In an exemplary embodiment, if the current step of the surgical procedure requires the surgeon to be looking at the greater curvature, but the surgeon is currently looking at a corneal notch, the system can overlay a directional arrow indicating the direction the surgeon needs to move in to reach the greater curvature.

[0102] In one embodiment, the system may overlay information about known parameters or characteristics of an object. In an exemplary embodiment, the system may identify a green surgical staple cartridge in a live stream. In the case of positive identification, the system may overlay parameters such as staple size, staple material, tissue thickness intended for use with the identified staple cartridge, and combinations thereof, onto the live stream associated with the identified staple cartridge.

[0103] In one embodiment, the system can overlay information on the display according to the identified orientation of an object identified in the live stream. In an exemplary embodiment, the system can identify an object in the display based on a comparison of the object with data associated with the object stored in memory. In one embodiment, the system can identify that an object is viewed in a first orientation, such as a side view of the object, and trigger a first overlay adjustment. In another embodiment, the system can identify that an object is viewed in a second orientation, such as a top view of the object, and trigger a second overlay adjustment different from the first overlay adjustment. In one embodiment, the system can identify that an object is viewed in a first orientation, such as at an angle of 30 degrees to its upright position, and trigger a first overlay adjustment. In another embodiment, the system can identify that an object is viewed in a second orientation, such as at an angle of 15 degrees to its upright position, and trigger a second overlay adjustment different from the first overlay adjustment.

[0104] In one embodiment, the system may be equipped with an interactive sensor, and the trigger event may be a user interacting with the interactive sensor. In various embodiments, the interactive sensor may be an audible sensor, and the trigger event may be a system that identifies known sounds, words, phrases, etc., which can be stored in memory via the audible sensor. In an exemplary embodiment, a surgeon may say "refocus," and the system may detect the word via the audible sensor and update the overlaid information on the display based on the identified word. In various embodiments, the trigger event may be based on a predetermined motion captured by an imaging device. In one embodiment, the predetermined motion may be stored in memory and compared with the motion captured by the imaging device. In an exemplary embodiment, a surgeon may move the end effector of a surgical instrument in a circular motion, and the system may detect the circular motion in the live stream and update the overlaid information on the display based on the detected motion. In various embodiments, adjustments made by the system to the overlaid information according to the detected interaction may be stored in memory. In one exemplary embodiment, a surgeon might say "clear," and the system, based on data stored in memory, can determine that "clear" means the surgeon wants to remove all overlaid information on the display.

[0105] In some embodiments, the adjustments the system makes to the overlaid information in accordance with detected interactions may be based on identified steps of a surgical procedure. In various embodiments, a context-aware module, such as context-aware module 6006, can determine the steps of a surgical procedure being performed based on one or more inputs received by the system. Based on the user-provided interactions and the determined steps of the surgical procedure, the system can adjust the overlaid information on the display accordingly. In one exemplary embodiment, a surgeon may provide the system with an audible command, such as a sound. The system can determine, via the context-aware module, that a particular step of the surgical procedure is being performed. The system can compare the sound with sounds stored in memory. In one embodiment, the memory can store a variety of executable instructions to be performed based on both the detected sound and the determined steps of the surgical procedure. In one embodiment, a sound may trigger a first adjustment to the overlaid information for one determined step and a second adjustment to the overlaid information for a second determined step, the first and second adjustments being distinct. In various embodiments, an audible command can trigger the same adjustment to the overlaid information independently of the determined steps of the surgical procedure.

[0106] In one embodiment, the system may detect location-based triggers that cause overlaid information on a display to adjust. In various embodiments, the system may include various sensors and visualization systems, such as those described elsewhere herein, that can track and / or determine the location of various components and / or individuals related to a surgical procedure. In one embodiment, the system may utilize GPS to determine the location of various components and / or individuals. In one embodiment, the system may include a digital compass to determine the location of various components and / or individuals. In one embodiment, the system may include sensors (e.g., accelerometers) for measuring velocity and acceleration data to determine the location of various components and / or individuals. In one embodiment, components and individuals for tracking may include location sensors that can be tracked by the system. The location tracking techniques provided above can be used alone and in combination with each other for the purpose of identifying the location of components and / or individuals inside or outside the OR.

[0107] In an exemplary embodiment, a surgeon works through colorectal and sigmoid colectomy mobilization using a surgical cutting device, and the live stream can be viewed on a display. The system can detect when the end effector of the surgical cutting device is approaching the cutting point of the blood supply via any number of position tracking techniques, as referenced above. Based on the system's detection that the end effector is approaching or has reached the cutting point, the system can adjust the display to overlay information to assist in the next mobilization step. As an example, the system can overlay the location and direction of blood flow, as well as where the blood is supplied, based on input from a surgical visualization system to the system, thereby assisting in the visualization of the next steps in the procedure.

[0108] In various embodiments, the system can detect the location of an individual or group of individuals inside or outside the OR via any number of location tracking techniques, as referenced above, and adjust the overlaid information on the display based on their detected locations(s). In one embodiment, the system can monitor the location of an individual, such as a nurse in a hospital, having a display such as a wearable AR device 66, for example. While the preceding discussion is in the context of a wearable AR device, it should be understood that any other display described herein can be used as an alternative to achieve the same results. In various embodiments, instead of the AR device 66, the nurse may have, for example, a tablet, a mobile phone, or any other portable display.

[0109] In various embodiments, the system can detect the location of an individual with a portable device to any number of locations. In one embodiment, the system can detect when an individual is approaching or has arrived at a location and adjust the information overlaid on the AR device 66 accordingly. In an exemplary embodiment, when a nurse wearing the AR device 66 arrives at a location such as a storage room door, the system can overlay information on the lens of the AR device associated with the storage room. In one embodiment, the system can overlay which rooms are behind the door. In one embodiment, the system can overlay which surgical instruments are stored in the storage room. In one embodiment, the system can overlay information if the storage room contains equipment required for a surgical procedure, based on the detected step of the surgical procedure by the system. In various embodiments, the system can overlay any amount of information useful to the individual in order to retrieve a desired portion of the equipment for a surgical procedure. In one embodiment, the system can overlay information based on the detected step of the surgical procedure, such as directional information indicating where a particular portion of the equipment can be obtained to complete the step of the surgical procedure. In various embodiments, the system may overlay information based on user input, such as language commands querying whether a particular part of a device can be found at an identified location. Location information, for example, which devices can be found at that location, can be stored in memory.

[0110] In various embodiments, the system can determine a surgical procedure step that is being performed or will be performed soon, and adjust the overlaid information on the AR device accordingly. In an exemplary embodiment, the system can determine, via a context-aware module, that a surgical stapling step will be performed soon and that a particular type of staple cartridge will be required. The system can, for example, overlay on the nurse's AR device that a particular type of staple cartridge will soon be needed. The system can further overlay on the AR device information such as, for example, where the staple cartridge can be found, what the staple cartridge looks like, the model number of the staple cartridge, or any other appropriate identifying information to help the nurse retrieve the staple cartridge. The system can further overlay directional information on the AR device to help the nurse find the staple cartridge. In an exemplary embodiment, the system can overlay information about where the staple cartridge can be found, such as a room number, shelf number, bin number, or any other appropriate descriptive information about where the staple cartridge can be found. In an exemplary embodiment, the system can use location tracking technology such as GPS to overlay directional arrows onto the lens of an AR device to visually guide a nurse to a location where the staple cartridge can be retrieved. In one embodiment, the system can overlay highlights on important features to assist in the retrieval of the staple cartridge. In an exemplary embodiment, when the door to a storage room where the staple cartridge is stored comes into the field of view of the AR device, the system can highlight the door to inform the nurse that the staple cartridge can be found behind the highlighted door. Any combination of the embodiments referenced above can be used in combination with each other to help identify the location of desired equipment.

[0111] In various other embodiments, the AR device can adjust the overlaid information based on surgical procedures, determined surgical steps of a surgical procedure, surgeon preferences, user input such as physical or verbal input, or a combination thereof. In one exemplary embodiment, when a nurse enters a storage room wearing the AR device, the system can adjust the overlaid information to point to or highlight a part of the equipment based on surgical procedures, determined surgical steps of a surgical procedure, surgeon preferences, user input such as physical or verbal input, or a combination thereof. In one embodiment, the system can adjust the overlaid information to highlight a part of the equipment in the storage room that is currently missing from the OR required or to be required for a surgical procedure. In one embodiment, the system can adjust the overlaid information based on a verbal request from a nurse inquiring about the location of a particular part of equipment. Based on the request, the system can adjust the overlaid information accordingly. In one embodiment, the system can highlight the requested item more brightly or more strongly than other highlighted items in the storage room. In another exemplary embodiment, the system may not highlight everything except the requested part of equipment.

[0112] In various embodiments, the system can track the location of AR devices and modify the relevance of trigger events based on that location. In one embodiment, a first user may be at a first location wearing a first AR device, and a second user may be at a second location wearing a second AR device. In an exemplary embodiment, a trigger event can be detected that causes the system to adjust the information overlaid on it. The system can detect that the first user is associated with the trigger event, and that the second user is not. In one embodiment, the system can detect that the first user is within a certain distance where the trigger event occurred, and the second user is outside that distance. Based on the determination, the system can update the information overlaid on the first AR device, but not on the second AR device. In an exemplary embodiment, a surgeon may wear an AR device to perform a surgical procedure, and a nurse may wear an AR device to retrieve a part of an instrument. When the nurse arrives in the storage room (location-based trigger event), the system can adjust the information overlaid on the nurse's AR device while maintaining what is overlaid on the surgeon's AR device. This selective adjustment of overlaid information prevents the display from adjusting when the overlaid information may be of little or no value to a particular individual.

[0113] In various embodiments, the system can adjust the information overlaid on the display based on any number of trigger events, such as those detected by a visualization system, such as any number of visualization systems described herein. In one embodiment, the system can adjust the overlaid information based on a determination of who is holding a particular surgical device. In one embodiment, the system can adjust the overlaid information based on a particular surgical device that is within the field of view of a visualization system. In one embodiment, the system can adjust the overlaid information based on where the surgical device is located relative to the patient. In an exemplary embodiment, when a particular surgical device comes within a threshold distance of the patient, as determined by any number of inputs, e.g., a visualization system, a position sensor, or any other position tracking technique described herein, the system can adjust the display to overlay information related to the surgical device. In an exemplary embodiment, when a particular surgical device moves beyond the threshold distance of the patient, as determined by any number of inputs, e.g., a visualization system, a position sensor, or any other position tracking technique described herein, the system can adjust the display to remove the overlaid information related to the surgical device. In one exemplary embodiment, when a particular surgical device reaches a threshold distance from the patient, as determined by any number of inputs, such as a visualization system, a position sensor, or any other position tracking technique described herein, the system can adjust the display to add overlaid information related to the surgical device.

[0114] In various embodiments, the system can adjust the information overlaid on the display based on determined priorities for surgical tasks. In one embodiment, the system can determine the steps of a surgical procedure, for example using a context-aware module, and adjust the importance or occurrence of trigger events based on the determination. In an exemplary embodiment, the system can use a context-aware module to determine that a surgical stapling step is being performed or is scheduled to be performed. The system can monitor trigger events during the surgical stapling step and determine whether adjustments to the overlaid information are needed according to their determined relevance to the surgical stapling step. In one embodiment, a trigger event such as excessive force being applied to the tissue being stapled can be detected. The system can determine that the excessive force is related to the current step of the surgical procedure and update the overlaid information on the display accordingly. In one embodiment, a trigger event such as the tissue temperature exceeding a temperature threshold can be detected. The system can determine that the excessive temperature is of little relevance to the current step of the surgical procedure and, based on the determination, can choose not to update the overlaid information. In various embodiments, the relevance of trigger events to surgical procedure steps can be stored in memory, defined and used, based on industry standards, or a combination thereof. In one embodiment, when the system determines that information is of low relevance to the current step of the surgical procedure, the system can overlay the information on the display, but can adjust how much of the display is overlaid. In one example, if the system detects a low-relevance trigger event while a surgical step is currently being performed, the system can overlay the information associated with the step on the display, but overlays only 50% of the size that the overlaid information would normally occupy.In other embodiments, the system may overlay information associated with the steps onto the display, or it may place the information in areas of the display that are not easily visible, such as the corners or edges of the display.

[0115] In various embodiments, the system can adjust the information overlaid on the display based on the importance of the data to the user operating the surgical device. In one embodiment, a surgeon can staple tissue using a surgical stapler. Based on data stored in memory, the system can detect excessive force applied to the tissue that the system deems important and adjust the display associated with the surgeon, such as an AR device 66, so that the detection of excessive force is notified to the surgeon using the surgical stapler.

[0116] In various embodiments, the system can adjust information overlaid on the display based on the detection of a specific type of surgical device used by the user. In one embodiment, the system can adjust the overlaid information to inform the user of problems associated with the particular surgical device being used, so that the user can be guided to identify potential points of failure. For example, the system can adjust the overlaid information to inform the user how potential misuse of a surgical device may cause secondary failures, such as failures of other surgical devices. In various embodiments, this data may be stored in memory. In various embodiments, this data may be accessible from a cloud-based system, such as cloud-based system 4.

[0117] In various embodiments, the system can adjust overlaid information on a display by moving information from a first display to a second display. In one embodiment, the system can detect the occurrence of a trigger event that may cause a change in the overlaid information on the primary display in the OR. In various embodiments, this change in the overlaid information may be changing the size of some of the information, the weight of some of the information, the position of some of the information, removing overlaid information, adding overlaid information, or a combination thereof. In one embodiment, as a result of the adjustment, the system can move information that is considered less relevant, such as less relevant to a particular surgical step being performed, from the first display to the second display, thereby maintaining information available to the surgical staff but not on a display that may not be the primary focus of the surgical staff.

[0118] In various embodiments, the system can adjust the overlaid information on the display based on the detection that a trigger event was induced by a surgical instrument used by a particular user. In some embodiments, the system can determine which surgical device is being actively used by which surgical personnel based on data received from sensors, modules, and / or visualization systems within the OR. In an exemplary embodiment, the system can determine that an energy device is being actively used based on data received from a generator module 40. In an exemplary embodiment, the system can determine that a surgical device is being actively used based on data received from a sensor module 29. In an exemplary embodiment, the system can determine that a surgical device is being actively used based on data received from an imaging module 25 or any number of visualization systems as described elsewhere in this specification. In an exemplary embodiment, the system can determine that a surgical device is being actively used based on inferences made from a situation awareness module. In one exemplary embodiment, the system may determine that a device is actively in use based on a system that receives signals indicating pairing occurring between a user-attached identifier and a surgical instrument, as described in U.S. Patent No. 10,758,310, which is incorporated herein by reference in whole. In various embodiments, the system may determine which surgical device is actively in use based on various sensors, modules, and input devices described herein, either individually or in combination with each other.

[0119] In various embodiments, the system can detect trigger events originating from surgical instruments actively controlled by the user and update overlaid information on the display accordingly. In one exemplary embodiment, the system can detect that a surgeon is actively using a tissue manipulator to manipulate tissue at the surgical site. The system can detect tissue tension exceeding a tissue tension threshold and determine that the tension was induced by a tissue manipulator associated with the surgeon. Based on the detected events and instrument occurrences, the system can adjust overlaid information on a display, such as a wearable AR device worn by the surgeon.

[0120] In various embodiments, the system can detect trigger events originating from outside of user-controlled active surgical instruments and update overlaid information on a display accordingly. In one exemplary embodiment, while a surgeon is actively using two instruments to incise the liver, a liver retractor not associated with the surgeon can be deployed and fixed to the liver. Based on the interaction of the two instruments actively used by the surgeon, tissue tension within the liver may be induced due to the fixed retractor exceeding a tension threshold. The system can detect the tissue tension induced by the retractor, for example, using a visualization system, and adjust overlaid information on a display, such as an AR device worn by the surgeon, even though the tissue tension event is induced by a component not associated with the surgeon. Thus, the system can update information on the AR device according to events induced by instruments or actions associated with or not associated with a particular user.

[0121] In various embodiments, the system can adjust the information overlaid on the display based on the detection of a risk event. In one embodiment, a risk event may be an event that has at least some possibility of causing an undesirable outcome with respect to a surgical procedure. In an exemplary embodiment, a risk event may be the detection of a particular type of device being used for a particular step of a surgical procedure. In another exemplary embodiment, a risk event may be the end effector of a surgical instrument that enters within a threshold distance of a critical structure such as an artery, vein, or tumor in a patient, for example. In an exemplary embodiment, a risk event may be a system that detects that a particular staple cartridge has been placed in an inappropriate stapling device for a determined step of a surgical procedure. In some embodiments, a risk event may be the end effector of a surgical instrument that articulates far away from its intended position. In any embodiment, the detection of a risk event may cause the system to overlay a warning or corrective step on the display that describes the detected risk event and possible corrective actions to avoid the risk event.

[0122] In various embodiments, the system can adjust the information overlaid on the display based on the detection of events originating outside the field of view. In one embodiment, an imaging device can capture a live stream of the surgical field and transmit the live stream to a display for surgical staff to view. Events may be triggered or occur outside the live stream of the surgical field, requiring attention, or reaction, or the surgeon. In an exemplary embodiment, the surgeon can manipulate tissue visible within the surgical field on the display. As a result of tissue manipulation, a portion of the tissue outside the surgical field may rupture as a result of threshold tension being inadvertently applied to the tissue. The system can detect the rupture, for example, via a visualization system, and alert the surgeon that the tissue rupture occurred outside the surgeon's field of view. This allows the surgeon to reposition the imaging device to the location of the rupture and take appropriate action. In one embodiment, the system can overlay directional information that informs the surgeon where to look to find events occurring outside the field of view.

[0123] In various embodiments, the system can detect trigger events that may cause adjustments to overlaid information on a wearable display. In some embodiments, in addition to adjusting overlaid information on a wearable display, the system can also adjust overlaid information on various other displays within the OR so that individuals not wearing the wearable AR device can also see the adjusted overlay. In one exemplary embodiment, the system can detect trigger events that cause adjustments to overlaid information on an AR device. In addition, the system can adjust overlaid information on other displays within the OR, such as display 19, for viewing by surgical personnel not wearing the AR device 66.

[0124] In various embodiments, the system can adjust the overlaid information on the display in any number of ways. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about the device's state in accordance with any number of sensors. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about the device's heat or thermal profile, which may be detected by a temperature sensor. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about the direction in which the imaging device should be moved to adjust its surgical field. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about hazardous areas, such as areas of tissue to be avoided to avoid potential injury to the patient. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about the tissue's state as determined by any number of sensors. In an exemplary embodiment, when a trigger event occurs, the system can overlay information about external implants detected by the system, such as clips or staples implanted in the patient.

[0125] In various embodiments, the system can detect the occurrence of a package containing open surgical components and adjust the overlaid information on the display accordingly. In one exemplary embodiment, the system can detect that a package containing a surgical staple cartridge reload has been opened. The system can overlay information associated with the components within the package, such as suggestions for a secondary package, such as a staple holder. In some embodiments, the system can overlay information on the display such as parameters associated with the components within the package, such as staple size, staple material, or the thickness of the tissue to be stapled by the cartridge.

[0126] In various embodiments, the system can detect surgical steps in a surgical procedure and adjust the overlaid information accordingly. In one embodiment, the system can detect surgical steps that require subsequent disposal steps, such as disposing of an old surgical staple cartridge in an instrument and replacing the instrument with a new cartridge. In an exemplary embodiment, the system can detect the completion of a stapling stroke and overlay an instruction to the surgical staff that the cartridge needs to be removed and replaced. In one embodiment, the overlaid information can further identify the type of replacement cartridge to be used to complete a subsequent stapling stroke. In various embodiments, the system can overlay information about where the surgical staple cartridge may be disposed of.

[0127] In various embodiments, the system can adjust overlaid information based on monitored parameters associated with a patient that reach or exceed a parameter threshold. In one embodiment, any number of sensors or systems can monitor patient-associated parameters, such as heart rate, and adjust the overlaid information accordingly. In one embodiment, the system can monitor the values ​​of various parameters and compare them to parameter thresholds stored in memory. If a parameter value reaches or exceeds a parameter threshold, the system can adjust the overlaid information to notify the user that the threshold has been reached or exceeded, so that subsequent actions can be taken. In some embodiments, the system can overlay corrective actions on the display that can help reduce the parameter value below the parameter threshold. In various embodiments, the system can monitor the rate of change of patient-associated parameters and adjust the overlaid information based on whether the rate of change reaches or exceeds a rate of change threshold.

[0128] In various embodiments, the system can detect the occurrence of a trigger event by detecting an accumulation of partial trigger events and comparing the accumulated events to a trigger event threshold. In one embodiment, the system can set a trigger event count and count the number of times a partial trigger event occurs. In an exemplary embodiment, the system can set the trigger event count to 0 at the start of a surgical procedure. In another exemplary embodiment, the system can set the trigger event count to 0 at the start of a particular step of a surgical procedure. In one embodiment, the system can reset the trigger event count to 0 at the end of a completed step of a surgical procedure. In some embodiments, the trigger event count can be set to a non-zero value such as 1, 2, 3, or any other appropriate integer. In various embodiments, the system can set the trigger event count based on user input, input from a situation awareness module based on detected steps of a surgical procedure, or a combination thereof.

[0129] In one embodiment, the system can detect partial trigger events, adjust the trigger event count based on the occurrence of partial trigger events, and adjust overlaid information on the display based on whether the trigger event count reaches or exceeds a trigger event threshold. In an exemplary embodiment, the system can set the trigger event threshold to three partial trigger events. The system can set the trigger even count to 0 at the start of a detected tissue manipulation step. The system can detect tissue tension induced in manipulated tissue against a tension threshold and can increment the trigger event count by 1 each time the tension threshold is reached or exceeded. If the trigger event count reaches or exceeds the trigger event threshold, the system can adjust the overlaid information on the display accordingly, for example, by issuing a warning to surgical staff or by providing corrective actions to ensure that the tension threshold is not reached or exceeded.

[0130] The ability to detect and count partial trigger events allows the system to track events that may be minor or insignificant individually, but whose accumulation could lead to a major or significant event. For example, in the tissue manipulation steps referenced above, inducing tissue tension exceeding the tension threshold on a single occasion may not excessively damage patient tissue, but multiple occurrences could result in rupture or tissue damage.

[0131] In various embodiments, a partial trigger event may include an additional trigger event that adds to the trigger event count and a negative trigger event that subtracts from the trigger event count. In the exemplary embodiment described above relating to tissue tension induced by a tissue manipulator, the tissue tension induced by the tissue manipulator may be an additional trigger event that adds 1 to the trigger event count. In some embodiments, the system may track the amount of time elapsed since the occurrence of the last additional trigger event and compare the elapsed time to a threshold time. If another additional trigger event is not induced by the time the elapsed time is used to look up the threshold time, the system may detect this as a negative trigger event and subtract 1 from the trigger event count. In various embodiments, a negative trigger event may be any event that reduces or eliminates the effect caused by an additional trigger event. In one exemplary embodiment, an additional trigger event may be reaching or exceeding a temperature threshold, and a negative trigger event may be applying a temperature below a temperature threshold to cool heated tissue. In other exemplary embodiments, a negative trigger event may be the administration of a drug, such as an injection, that counteracts or eliminates the effects caused by a positive trigger event.

[0132] In various embodiments, additional and negative partial trigger events may have different weights. In one embodiment, a first type of additional trigger event may add 1 to the trigger event count, while a second type of additional trigger event may add 2 to the trigger event count. In one embodiment, a first type of negative trigger event may subtract 1 from the trigger event count, while a second type of negative trigger event may subtract 2 from the trigger event count. Any number of weights, such as 1 to n, can be assigned to partial trigger events, where n is the trigger event threshold (i.e., an additional trigger event count with weight n brings the trigger event threshold to the limit when it occurs). The weights may be user-defined, stored in memory, based on industry standards, or a combination thereof. In various embodiments, partial trigger events may be non-integer values, such as 0.5, 1.5, or 1.8, for example.

[0133] The ability to add to and subtract from the trigger event count allows the system to track events that may be minor or insignificant individually, but whose cumulative effect could lead to a major or significant event (additional trigger events). However, the system can also detect events that minimize or reduce additional trigger events and therefore can be removed from the trigger event count (negative trigger events). For example, in the tissue manipulation step referenced above, inducing tissue tension exceeding the tension threshold individually may not excessively damage patient tissue, but multiple occurrences can result in rupture or damaged tissue (additional trigger events). However, during the course of a surgical procedure that may last for several hours, exceeding the tension threshold may be expected to occur more times than the trigger event threshold. However, this number of occurrences over a long period may not cause significant harm to the tissue. Therefore, the system can subtract from the trigger event count to keep the trigger event count below the trigger event threshold and to prevent the overlaid information from being adjusted when the overlaid information is not needed (negative trigger events).

[0134] In various embodiments, the system can detect cancel trigger events that can reset the trigger event count. In one embodiment, the system can detect the number of times a tension exceeding a tension threshold is induced in the tissue during a surgical procedure step. The system can detect that the current tissue manipulation step of the surgical procedure has ended and that a new step of the surgical procedure is being performed. Thus, the system can detect the completion of the tissue manipulation step as a cancel trigger event that resets the trigger event count, such as by resetting the count back to 0.

[0135] In various embodiments, the system can monitor multiple trigger events that may have different trigger event thresholds. In one embodiment, a first trigger event may have a first trigger event threshold, such as the system detecting three partial trigger events, and a second trigger event may have a second trigger event threshold, such as the system detecting four partial trigger events. In one embodiment, having different trigger event thresholds allows the system to monitor partial trigger events that may have varying degrees of severity.

[0136] In various embodiments, additional trigger events may be identical or similar additional trigger events. In one exemplary embodiment, the system may reach a trigger event threshold when it detects the occurrence of the same three partial trigger events, such as the tension of the tissue reaching or exceeding a tension threshold. This allows the system to monitor a specific type of event associated with the trigger event threshold and adjust the overlaid information when that specific type of event occurs a threshold number of times.

[0137] In various embodiments, the additional trigger events may be different additional trigger events. In one exemplary embodiment, the trigger event threshold can be reached when the system detects the occurrence of three different types of additional trigger events, such as tension induced in tissue reaching a tension threshold, force applied to tissue reaching a force threshold, and heat applied to tissue reaching a temperature threshold. This allows the system to monitor different events that may not be significant individually but, when combined, could damage tissue. Thus, the trigger event threshold can be reached when multiple independent partial trigger events occur, and therefore the system can be made to adjust the overlaid information on the display.

[0138] Figure 13 shows a flowchart illustrating the operation of an exemplary method 10000 for determining the display arrangement of competing surgical data in order to present it on a display such as display 6005 showing a live stream of the surgical field. In one embodiment, the live stream may be captured by an imaging device such as imaging device 6004 that images the surgical field. The imaging device may be operably connected to a control system such as system 6000, which is similarly operably connected to a display. The control system may transmit the live stream of the surgical field from the imaging device to the display so that surgical personnel can view the live stream on the display.

[0139] In various embodiments, method 10000 includes overlaying information associated with a surgical procedure onto a live stream (10005). In one embodiment, the control system may receive inputs from various modules, sensors, user input devices, and situation awareness modules, and overlay information associated with these inputs onto a display.

[0140] In various embodiments, method 10000 includes detecting the occurrence of a trigger event (10010). In one embodiment, the trigger event may be any number of trigger events described herein that may result in the system adjusting overlaid information on a display.

[0141] In various embodiments, method 10000 includes adjusting overlaid information (10015) based on the occurrence of a trigger event. In an exemplary embodiment, the adjustment may be that a control system overlays information on a display associated with the trigger event. Any number of adjustments to the overlaid information can be made as described herein.

[0142] Figure 14 shows a flowchart illustrating the operation of an exemplary method 10100 for determining the display arrangement of competing surgical data in order to present it on a display such as display 6005 showing a live stream of the surgical field. In one embodiment, the live stream may be captured by an imaging device such as imaging device 6004 that images the surgical field. The imaging device may be operably connected to a control system such as system 6000, which is similarly operably connected to a display. The control system may transmit the live stream of the surgical field from the imaging device to the display so that surgical personnel can view the live stream on the display.

[0143] In various embodiments, method 10100 includes overlaying information associated with a surgical procedure onto a live stream (10105). In one embodiment, the control system may receive inputs from various modules, sensors, user input devices, and situation awareness modules, for example, and overlay information associated with these inputs onto a display.

[0144] In various embodiments, method 10100 further includes setting a trigger event count (10110). In one embodiment, the system may set the trigger event count to a value such as 0 at the beginning or start of a step of a surgical procedure.

[0145] In various embodiments, method 10100 further includes detecting partial trigger events (10115). In one embodiment, the system can detect partial trigger events, such as, for example, tissue tension reaching or exceeding a tension threshold, tissue temperature reaching or exceeding a temperature threshold, or force applied to tissue reaching or exceeding a temperature threshold. The system can detect partial trigger events using any number of sensors, modules, imaging systems, or combinations thereof, as described elsewhere in this specification.

[0146] In various embodiments, method 10100 further includes adjusting the trigger event count based on the occurrence of partial trigger events (10120). In one embodiment, the system may index the trigger event count to increment by 1 when an additional trigger event is detected. In one embodiment, the system may index the trigger event count to decrement by 1 when a negative trigger event is detected. In one embodiment, the system may reset the trigger event count back to an initial trigger event count such as 0 when a cancellation trigger event is detected.

[0147] In various embodiments, method 10100 further includes adjusting the overlaid information (10125) based on whether the trigger event count reaches or exceeds a trigger event threshold. In one embodiment, the system may compare the trigger event count to a trigger event threshold and adjust the overlaid information based on the determination that the trigger event threshold has been reached or exceeded. In one embodiment, if the trigger event threshold has been reached or exceeded, the system may overlay a warning on the display indicating that the trigger event threshold has been reached or exceeded.

[0148] Figure 15 shows a flowchart illustrating the operation of an exemplary method 10100 for determining the display arrangement of competing surgical data in order to present it on a display such as display 6005 showing a live stream of the surgical field. In one embodiment, the live stream may be captured by an imaging device such as imaging device 6004 that images the surgical field. The imaging device may be operably connected to a control system such as system 6000, which is similarly operably connected to a display. The control system may transmit the live stream of the surgical field from the imaging device to the display so that surgical personnel can view the live stream on the display.

[0149] In various embodiments, method 10200 includes overlaying information associated with a surgical procedure onto a live stream (10205). In one embodiment, the control system may receive inputs from various modules, sensors, user input devices, and situation awareness modules, and overlay information associated with these inputs onto a display.

[0150] In various embodiments, method 10200 further includes determining the steps of a surgical procedure (10210). In one embodiment, the system can determine the steps of a surgical procedure, such as the steps currently being performed or the steps to be performed soon, based on any number of inputs provided to the system. In one embodiment, the system can determine the steps of a surgical procedure using a situational awareness module that can receive various amounts of information from sensors, modules, and devices for the purpose of determining the steps of a surgical procedure.

[0151] In various embodiments, method 10200 further includes detecting the occurrence of a trigger event (10215). In one embodiment, the trigger event may be any number of trigger events described herein that may result in the system adjusting overlaid information on a display.

[0152] In various embodiments, method 10200 further includes adjusting the overlaid information based on the occurrence of a trigger event and the steps of the surgical procedure (10120). In one exemplary embodiment, the system may determine that the next step of the surgical procedure is a surgical stapling operation and the trigger event is the completion of the surgical procedure step. The system may adjust the display to overlay information about the next step of the surgical procedure, such as the type of staple cartridge required to complete the surgical procedure step. In another exemplary embodiment, the system may determine that the current step of the surgical procedure is a tissue manipulation step and the trigger event is a force applied to tissue that reaches or exceeds a force threshold. The system may adjust the display to overlay information related to reaching or exceeding the threshold, such as informing the user that less pressure should be applied.

[0153] Various additional aspects of the subject matter described herein are illustrated in the following numbered examples.

[0154] Example 1: A surgical system comprising: an imaging device; a display configured to show a live stream of the surgical field of a surgical procedure, the display of which the live stream is captured by the imaging device; and a control system operably coupled to the imaging device and the display, the control system configured to overlay information associated with the surgical procedure onto the live stream, detect the occurrence of a trigger event, and adjust the overlaid information based on the occurrence of the trigger event.

[0155] Example 2: The surgical system according to Example 1, wherein the control system is configured to monitor parameters associated with a patient on whom a surgical procedure is being performed, and trigger events include the parameter value reaching or exceeding a parameter threshold.

[0156] Example 3: The surgical system according to Example 1 or 2, wherein the control circuit is configured to monitor a parameter associated with a patient on whom a surgical procedure is being performed, and the trigger event includes the rate of change of the parameter value reaching or exceeding a rate of change threshold.

[0157] Example 4: The surgical system according to any one of Examples 1 to 3, further comprising a surgical device associated with a surgeon, wherein the trigger event includes an event induced by the surgical device.

[0158] Example 5: The surgical system according to any one of Examples 1 to 4, further comprising a first surgical device associated with a surgeon and a second surgical device not associated with a surgeon, wherein the trigger event includes an event induced by the second surgical device.

[0159] Example 6: A surgical system according to any one of Examples 1 to 5, wherein the trigger event includes a trigger event threshold, and the control system is further configured to set a trigger event count, detect partial trigger events, and adjust the trigger event count based on the occurrence of partial trigger events, and detecting the occurrence of a trigger event includes detecting that the trigger event count has reached or exceeded the trigger event threshold.

[0160] Example 7: The surgical system according to Example 6, wherein a partial trigger event includes an additional trigger event and a negative trigger event, and adjusting the trigger event count includes adding to the trigger event count based on the occurrence of an additional trigger event and subtracting from the trigger event count based on the occurrence of a negative trigger event.

[0161] Example 8: A surgical system according to any one of Examples 1 to 7, further comprising a memory for storing data associated with an object, wherein the control system is further configured to identify an object in a live stream and compare the object with data associated with the object stored in memory, and a trigger event includes the identification of the object from the comparison.

[0162] Example 9: The surgical system according to any one of Examples 1 to 8, further comprising an interactive sensor, wherein a trigger event includes the user interacting with the interactive sensor.

[0163] Example 10: The surgical system according to any one of Examples 1 to 9, further comprising a surgical instrument including an end effector, wherein a control system is further configured to monitor the position of the end effector, and a trigger event includes the end effector reaching a certain location.

[0164] Example 11: A surgical system according to any one of Examples 1 to 10, wherein the control system is configured to monitor the user's location, and a trigger event includes the user reaching a certain location.

[0165] Example 12: The surgical system according to Example 11, wherein the adjustment of overlaid information includes overlaying location-related information.

[0166] Example 13: The surgical system according to Example 12, wherein the overlaid information associated with a location includes information associated with a surgical procedure.

[0167] Example 14: The surgical system according to Example 12 or 13, further comprising a situation awareness module operably connected to a control system, wherein the control system is further configured to determine the steps of a surgical procedure by the situation awareness module, and the location-associated overlaid information includes information associated with the steps of a surgical procedure.

[0168] Example 15: A surgical system according to any one of Examples 12-14, wherein the overlaid information associated with a location includes information based on user input.

[0169] Example 16: A surgical system according to any one of Examples 1 to 15, further comprising a situation awareness module operably coupled to a control system, wherein the control system is further configured by the situation awareness module to determine steps of a surgical procedure and to determine components required to perform the steps of the surgical procedure, the trigger event includes the control system determining the components required to perform the steps of the surgical procedure, and the adjustment of overlaid information includes providing directional information about where the components may be located.

[0170] Example 17: The surgical system according to any one of Examples 1 to 16, wherein the display includes a first display, the surgical system further includes a second display, and adjusting the overlaid information includes moving the overlaid information from the first display to the second display.

[0171] Example 18: A surgical system comprising: an imaging device; a display configured to show a live stream of the surgical field of a surgical procedure, the display of which the live stream is captured by the imaging device; and a control system operably coupled to the imaging device and the display, the control system configured to overlay information associated with the surgical procedure onto the live stream, set a trigger event count, detect partial trigger events, adjust the trigger event count based on the occurrence of partial trigger events, and adjust the overlaid information based on whether the trigger event count reaches or exceeds a trigger event threshold.

[0172] Example 19: The surgical system according to Example 18, wherein a partial trigger event includes an additional trigger event and a negative trigger event, and adjusting the trigger event count includes adding to the trigger event count based on the occurrence of an additional trigger event and subtracting from the trigger event count based on the occurrence of a negative trigger event.

[0173] Example 20: A surgical system comprising: an imaging device; a display configured to show a live stream of the surgical field of a surgical procedure, the display being captured by the imaging device; a situation awareness module; and a control system operably connected to the imaging device, the display, and the situation awareness module, the control system being configured to overlay information associated with the surgical procedure onto the live stream, and to determine the steps of the surgical procedure, detect the occurrence of a trigger event, and adjust the overlaid information based on the occurrence of the trigger event and the steps of the surgical procedure, by the situation awareness module.

[0174] While several forms have been shown and described, it is not the applicant's intention to limit or restrict the attached claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be implemented and will be conceived by those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element related to the described form can be alternatively described as a means for providing the function performed by that element. Also, while materials are disclosed with respect to specific components, other materials may be used. Therefore, it should be understood that the above description and the attached claims are intended to cover all such modifications, combinations, and variations as being included within the scope of the disclosed forms. The attached claims are intended to cover all such modifications, variations, alterations, substitutions, alterations, and equivalents.

[0175] The detailed descriptions above have described various forms of apparatus and / or processes using block diagrams, flowcharts and / or embodiments. To the extent that such block diagrams, flowcharts and / or embodiments include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flowcharts and / or embodiments can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some or all of the forms disclosed herein can be equivalently implemented on integrated circuits as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code falls within the scope of the skills of those skilled in the art in light of this disclosure. Furthermore, as will be understood by those skilled in the art, the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and the specific forms of the subject matter described herein are applicable regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0176] Instructions used to program logic to implement various disclosed embodiments may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, instructions may be distributed over a network or by other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but are not limited to floppy diskettes, optical disks, compact disks, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage used for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, non-temporary computer-readable media may include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0177] When used in any aspect of this specification, the term “control circuit” can mean, for example, hardwired circuits, programmable circuits (e.g., computer processors, processing units, processors, microcontrollers, microcontroller units, controllers, digital signal processors (DSPs), programmable logic devices (PLDs), programmable logic arrays (PLAs), or field-programmable gate arrays (FPGAs) including one or more individual instruction processing cores), state-machine circuits, firmware that stores instructions executed by programmable circuits, and any combination thereof. Control circuits can be embodied collectively or individually as circuits that form part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, or a smartphone. Accordingly, as used herein, “control circuit” includes, but is not limited to, an electrical circuit having at least one separate electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application-specific integrated circuit, an electrical circuit forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program that performs at least partially the processes and / or devices described herein, or a microprocessor configured by a computer program that performs at least partially the processes and / or devices described herein), an electrical circuit forming a memory device (e.g., in the form of random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electric installation). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog form, digital form, or some combination thereof.

[0178] When used in any aspect of this specification, the term “logic” may mean an application, software, firmware, and / or circuit configured to perform any of the operations described above. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-temporary computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and / or hardcoded (e.g., non-volatile) data in a memory device.

[0179] When used in any aspect of this specification, the terms “component,” “system,” “module,” etc., may refer to a control circuit, a computer-related entity, hardware, a combination of hardware and software, software, or running software.

[0180] Where used in any aspect of this specification, “algorithm” means a self-consistent sequence of steps leading to a desired result, and “step” means the manipulation of physical quantities and / or logical states that can take the form of electrical or magnetic signals, which are not necessarily required but can be stored, transferred, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities, or simply are convenient labels applied to these quantities and / or states.

[0181] A packet-switched network is one example of a network. Communication devices can communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol is the Ethernet communication protocol, which can enable communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of the Ethernet standard. Alternatively or additionally, communication devices can communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices can communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol conforms to or may be compatible with standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol conforms to or may be compatible with the ATM standard and / or later versions of this standard, published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0". Naturally, different and / or later developed connection-oriented network communication protocols are equally construed herein.

[0182] Unless otherwise explicitly stated, as is evident from the foregoing disclosures, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the foregoing disclosures should be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage, transmission, or display device.

[0183] One or more components may be referred to herein as “configured to,” “configurable to,” “operable / operative to,” “adapted / adaptable,” “able to,” “conformable / conformed to,” and so on. Those skilled in the art will understand that “configured to” generally encompasses active components and / or inactive components and / or standby components, unless the context should interpret it otherwise.

[0184] The terms “proximal” and “distal” are used herein in reference to the clinician operating the handle portion of a surgical instrument. “Proximal” refers to the part closest to the clinician, and “distal” refers to the part further away from the clinician. For convenience and clarity, spatial terms such as “vertical,” “horizontal,” “up,” and “down” may be used herein in reference to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0185] Those skilled in the art will generally understand that the terms used herein, and especially in the appended claims (e.g., the text of the appended claims), are generally intended to be "open" terms (for example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "includes" should be interpreted as "includes but is not limited to"). Furthermore, those skilled in the art will understand that if a particular number is intended in an introduced claim recitation, such intent is clearly stated in the claim, and if such statement is not present, such intent does not exist. For example, to aid understanding, subsequent appended claims may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be interpreted as suggesting that any particular claim containing such introduced claim description is limited to claims containing only one such description, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" (for example, "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using a definite article.

[0186] In addition, even if a specific number is explicitly stated in the introduced claim, it will be recognized by those skilled in the art that such a statement should typically be interpreted as meaning at least the number stated (for example, if there is a statement that is simply “two descriptions” without any other modifiers, it generally means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to “at least one of A, B, and C, etc.” is used, such a notation is generally intended to be understood in a way that those skilled in the art will understand (for example, “a system having at least one of A, B, and C” is not limited to systems having only A, only B, only C, both A and B, both A and C, both B and C and / or all of A, B and C, etc.). When expressions similar to "at least one of A, B, or C" are used, such expressions are generally intended to be understood in a way that a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Furthermore, a person skilled in the art will understand that, typically, any disjunctive word and / or phrase representing two or more selective terms should be understood, whether in the specification, claims, or drawings, as intended to include the possibility of including one of those terms, any of those terms, or both of those terms, unless the context requires a different interpretation. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0187] With respect to the attached claims, those skilled in the art will understand that the operations cited herein may generally be performed in any order. Furthermore, while various operations are shown in sequence(s), it should be understood that the operations may be performed in any order other than those shown, or simultaneously. Examples of such alternative orderings may include repetition, alternation, interruption, reordering, augmentation, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context should imply otherwise. Moreover, terms such as “responsive to,” “related to,” or other past tense adjectives are generally not intended to exclude such variations, unless the context should imply otherwise.

[0188] It is worth noting that any reference to “one aspect,” “aspect,” “example,” or “example” means that the specific feature, structure, or characteristic described in relation to that aspect is included in at least one aspect. Therefore, the phrases “in one aspect,” “in aspect,” “example,” and “example” found in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, specific features, structures, or characteristics can be combined in any preferred manner in one or more aspects.

[0189] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material does not conflict with this Specified. Disclosures expressly stated herein, both in themselves and to the extent required, shall supersede any conflicting statements incorporated herein by reference. Any material, or any part thereof, that is referred to as being incorporated herein by reference but conflicts with current definitions, views, or other disclosures contained herein shall be incorporated only to the extent that there is no conflict between the incorporated material and the current disclosures.

[0190] In summary, the numerous benefits that can be obtained as a result of using the concepts described herein have been described. The above descriptions of one or more forms are presented for illustrative and explanatory purposes only. They are not intended to be comprehensive or to be limited to the exact forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms have been selected and described to illustrate the principle and practical applications, thereby enabling a person skilled in the art to utilize the various forms, along with various modifications, for specific conceivable uses. The claims presented herein are intended to define the overall scope.

[0191] [Implementation Method] (1) A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, A control system operably connected to the imaging device and the display, The information associated with the surgical procedure is overlaid on the live stream. Detect the occurrence of a trigger event, A surgical system comprising: a control system configured to adjust the overlaid information based on the occurrence of the trigger event. (2) The surgical system according to Embodiment 1, wherein the control system is configured to monitor a parameter associated with a patient on whom the surgical procedure is being performed, and the trigger event includes the value of the parameter reaching or exceeding a parameter threshold. (3) The surgical system according to Embodiment 1, wherein a control circuit is configured to monitor a parameter associated with a patient on whom the surgical procedure is being performed, and the trigger event includes the rate of change of the value of the parameter reaching or exceeding a rate of change threshold. (4) The surgical system according to Embodiment 1, further comprising a surgical device associated with a surgeon, wherein the trigger event includes an event induced by the surgical device. (5) A first surgical device associated with the surgeon, The device further comprises a second surgical device not associated with the surgeon, The surgical system according to Embodiment 1, wherein the trigger event includes an event induced by the second surgical device.

[0192] (6) The trigger event includes a trigger event threshold, and the control system Set the trigger event count, Detect partial trigger events, It is further configured to adjust the trigger event count based on the occurrence of partial trigger events, The surgical system according to Embodiment 1, wherein detecting the occurrence of the trigger event includes detecting that the trigger event count reaches or exceeds the trigger event threshold. (7) The partial trigger event includes an additional trigger event and a negative trigger event, and the trigger event count is adjusted accordingly. The trigger event count is increased based on the occurrence of additional trigger events, A surgical system according to Embodiment 6, comprising subtracting from the trigger event count based on the occurrence of a negative trigger event. (8) The control system further comprises a memory for storing data associated with an object, Identify the objects in the live stream, The object is further configured to compare with the data associated with the object stored in the memory, The surgical system according to Embodiment 1, wherein the trigger event includes the identification of the object from the comparison. (9) The surgical system according to Embodiment 1, further comprising an interactive sensor, wherein the trigger event includes a user interacting with the interactive sensor. (10) The surgical system according to Embodiment 1, further comprising a surgical instrument including an end effector, wherein the control system is further configured to monitor the position of the end effector, and the trigger event includes the end effector reaching a certain location.

[0193] (11) The surgical system according to Embodiment 1, wherein the control system is configured to monitor the location of a user, and the trigger event includes the user reaching a certain location. (12) The surgical system according to Embodiment 11, wherein adjusting the overlaid information includes overlaying information associated with the location. (13) The surgical system according to Embodiment 12, wherein the overlaid information associated with the location includes information associated with the surgical procedure. (14) The surgical system according to Embodiment 12, further comprising a situation awareness module operably connected to the control system, wherein the control system is further configured by the situation awareness module to determine the steps of the surgical procedure, and the overlaid information associated with the location includes information associated with the steps of the surgical procedure. (15) The surgical system according to Embodiment 12, wherein the overlaid information associated with the location includes information based on user input.

[0194] (16) The control system further comprises a situation recognition module operably connected to the control system, and the control system The situation recognition module determines the steps of the surgical procedure, Further configured to determine the components required to perform the steps of the surgical procedure, The trigger event includes the control system determining the components necessary to perform the steps of the surgical procedure, The surgical system according to Embodiment 1, wherein adjusting the overlaid information provides directional information about where the components may be located. (17) The surgical system according to Embodiment 1, wherein the display includes a first display, and the surgical system further comprises a second display, and adjusting the overlaid information includes moving the overlaid information from the first display to the second display. (18) A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, A control system operably connected to the imaging device and the display, The information associated with the surgical procedure is overlaid on the live stream. Set the trigger event count, Detect partial trigger events, Based on the occurrence of partial trigger events, the trigger event count is adjusted. A surgical system comprising: a control system configured to adjust the overlaid information based on whether the trigger event count reaches or exceeds a trigger event threshold. (19) The partial trigger event includes an additional trigger event and a negative trigger event, and the trigger event count is adjusted accordingly. The trigger event count is increased based on the occurrence of additional trigger events, A surgical system according to embodiment 18, comprising subtracting from the trigger event count based on the occurrence of a negative trigger event. (20) A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, Situation awareness module, A control system operably connected to the imaging device, the display, and the situation recognition module, The information associated with the surgical procedure is overlaid on the live stream. The situation recognition module determines the steps of the surgical procedure, Detect the occurrence of a trigger event, A surgical system comprising: a control system configured to adjust the overlaid information based on the occurrence of the trigger event and the steps of the surgical procedure.

Claims

1. A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, A control system operably connected to the imaging device and the display, The information associated with the surgical procedure is overlaid on the live stream. Detect the occurrence of a trigger event, A control system configured to adjust the overlaid information based on the occurrence of the trigger event, The control system, It is further configured to set a trigger event count, detect the occurrence of a partial trigger event, and, based on that detection, add or subtract from the trigger event count. A surgical system in which detecting the occurrence of the trigger event includes detecting that the trigger event count reaches or exceeds a trigger event threshold.

2. The surgical system according to claim 1, wherein the control system is configured to monitor a parameter associated with a patient on whom the surgical procedure is being performed, and the trigger event includes the value of the parameter reaching or exceeding a parameter threshold.

3. The surgical system according to claim 1, wherein the control system is configured to monitor a parameter associated with a patient on whom the surgical procedure is being performed, and the trigger event includes the rate of change of the value of the parameter reaching or exceeding a rate of change threshold.

4. The surgical system according to claim 1, further comprising a surgical device associated with a surgeon, wherein the trigger event includes an event induced by the surgical device.

5. A first surgical device associated with the surgeon, The device further comprises a second surgical device not associated with the surgeon, The surgical system according to claim 1, wherein the trigger event includes an event induced by the second surgical device.

6. The aforementioned partial trigger event includes an additional trigger event which is an event that causes the trigger event count to increase, and a negative trigger event which is an event that causes the trigger event count to decrease, and the addition or subtraction of the trigger event count is performed as follows: The occurrence of the aforementioned additional trigger event is added to the trigger event count, The surgical system according to claim 1, further comprising subtracting from the trigger event count based on the occurrence of the negative trigger event.

7. The surgical system according to claim 6, wherein the negative trigger event includes the occurrence of a threshold time since the occurrence of the additional trigger event.

8. The control system further comprises memory for storing data associated with an object, The system is further configured to compare objects in the live stream with the data associated with the objects stored in memory, The surgical system according to claim 1, wherein the trigger event includes identifying the type or orientation of the object from the comparison.

9. The surgical system according to claim 1, further comprising an interactive sensor which is a user interface, wherein the trigger event is an input from a user to the interactive sensor.

10. The surgical system according to claim 1, further comprising a surgical instrument including an end effector, wherein the control system is further configured to monitor the position of the end effector, and the trigger event includes the end effector reaching a certain location.

11. The surgical system according to claim 1, wherein the control system is configured to monitor the user's location, and the trigger event includes the user reaching a certain location.

12. The surgical system according to claim 11, wherein adjusting the overlaid information includes overlaying information associated with the location.

13. The surgical system according to claim 12, wherein the overlaid information associated with the location includes information associated with the surgical procedure.

14. The surgical system according to claim 12, further comprising a situation awareness module operably connected to the control system, wherein the control system is further configured to determine the steps of the surgical procedure by the situation awareness module, and the overlaid information associated with the location includes information associated with the steps of the surgical procedure.

15. The surgical system according to claim 12, wherein the overlaid information associated with the location includes information based on user input.

16. The control system further comprises a situation recognition module operably connected to the control system, and the control system The situation recognition module determines the steps of the surgical procedure, Further configured to determine the components required to perform the steps of the surgical procedure, The trigger event includes the control system determining the components necessary to perform the steps of the surgical procedure, The surgical system according to claim 1, wherein adjusting the overlaid information provides directional information indicating where the components may be located.

17. The surgical system according to claim 1, wherein the display includes a first display, and the surgical system further comprises a second display, and adjusting the overlaid information includes moving the overlaid information from the first display to the second display.

18. A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, A control system operably connected to the imaging device and the display, The information associated with the surgical procedure is overlaid on the live stream. Set a trigger event count, detect the occurrence of a partial trigger event, and based on that detection, add or subtract from the trigger event count. A surgical system comprising: a control system configured to adjust the overlaid information based on whether the trigger event count reaches or exceeds a trigger event threshold.

19. The aforementioned partial trigger event includes an additional trigger event which is an event that causes the trigger event count to increase, and a negative trigger event which is an event that causes the trigger event count to decrease, and adjusting the trigger event count is as follows: The occurrence of the aforementioned additional trigger event is added to the trigger event count, The surgical system according to claim 18, further comprising subtracting from the trigger event count based on the occurrence of the negative trigger event.

20. A surgical system, Imaging device and A display configured to show a live stream of the surgical field during a surgical procedure, wherein the live stream is captured by the imaging device, Situation awareness module, A control system operably connected to the imaging device, the display, and the situation recognition module, The information associated with the surgical procedure is overlaid on the live stream. The situation recognition module determines the steps of the surgical procedure, Detect the occurrence of a trigger event, A control system configured to adjust the overlaid information based on the occurrence of the trigger event and the steps of the surgical procedure, The control system, It is further configured to set a trigger event count, detect the occurrence of a partial trigger event, and, based on that detection, add or subtract from the trigger event count. A surgical system in which detecting the occurrence of the trigger event includes detecting that the trigger event count reaches or exceeds a trigger event threshold.

Images