Customization of overlaid data and configuration

Abstract

A method of distributing data among members of a surgical team may include receiving, by a modular control tower, imaging data from a plurality of imaging devices, receiving, by the modular control tower, device dependent data from each of a plurality of intelligent surgical instruments, associating, by the modular control tower, a display device with a member of the surgical team, defining, by the modular control tower, functional roles of the members of the surgical team, and displaying, by the modular control tower, an augmented reality display with the display device. The augmented reality display on the display device may include virtual objects based on the imaging data, the device dependent data, the functional roles of the members of the surgical team, and surgical activities by the members of the surgical team. An interactive surgical system may implement this method.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 174,674, filed on April 14, 2021, entitled "HEADS UP DISPLAY", and U.S. Provisional Patent Application No. 63 / 284,326, filed on November 30, 2021, entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS", under 35 U.S.C. § 119(e). The disclosures of each of these applications are hereby incorporated by reference in their entirety into this specification.

Background Art

[0002] This disclosure relates to devices, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure, it would be desirable to provide an augmented reality interactive experience of the real - world environment in which objects that exist in the real world are enhanced by overlaying computer - generated perceptual information across multiple sensory modalities, including, at times, vision, hearing, touch, proprioception, and smell. In the context of this disclosure, images of the surgical field and surgical instruments and other objects that appear in the surgical field are enhanced by overlaying computer - generated visual, auditory, tactile, proprioceptive, olfactory, or other sensory information onto the real - world images of the surgical field and the instruments or other objects that appear in the surgical field. The images may be streamed in real - time or may be still images.

[0003] Real-world surgical instruments include a variety of surgical devices, including energy, staplers, or combinations of energy and staplers. Energy-based medical devices include, but are not limited to, radio frequency (RF) based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including obesity, thoracic, colorectal, obstetric and gynecological, urological, and general surgery. [Overview of the project] [Means for solving the problem]

[0004] In various examples, the Disclosure provides a method for distributing data among members of a surgical team, the method comprising: receiving imaging data from multiple imaging devices via a modular control tower; receiving device-dependent data from each of multiple intelligent surgical instruments via a modular control tower; associating display devices with members of the surgical team via a modular control tower; defining the functional roles of the members of the surgical team via a modular control tower; and displaying an augmented reality display via a display device via a modular control tower. The augmented reality display on the display device includes virtual objects based on the imaging data, device-dependent data, the functional roles of the members of the surgical team, and surgical activities performed by the members of the surgical team.

[0005] In another example, the disclosure provides an automated surgical system comprising a modular control tower, a plurality of imaging devices communicating with the modular control tower, a plurality of intelligent surgical instruments, and a plurality of display devices communicating with the modular control tower. Each of the plurality of display devices is associated by the modular control tower with one or more members of a surgical team, each of which is defined by a functional role. The modular control tower comprises a controller that communicates with one or more memory components configured to store instructions, which, when executed by the controller, cause the controller to receive imaging data from the plurality of imaging devices, receive device-dependent data from each of the plurality of intelligent surgical instruments, and display an augmented reality display on each of the plurality of display devices. The augmented reality display on a designated display device may include imaging data, device-dependent data, the functional role of a designated member of the surgical team associated with the designated display device, and virtual objects based on surgical activities performed by the designated member of the surgical team. [Brief explanation of the drawing]

[0006] The various embodiments described herein with respect to both configuration and operation methods, along with their further purposes and advantages, can be best understood by referring to the following description in conjunction with the accompanying drawings. [Figure 1] This is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 2] This is a diagram of a surgical system used to perform surgical procedures in an operating room, according to one aspect of the present disclosure. [Figure 3] One aspect of the present disclosure is a visualization system, a robotic system, and a surgical hub paired with an intelligent instrument. [Figure 4]This figure shows a surgical data network, according to one aspect of the present disclosure, which includes a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This figure shows a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 6] This figure shows a surgical hub, including a plurality of modules connected to a modular control tower, according to one aspect of the present disclosure. [Figure 7] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 8] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 9] This figure shows an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. [Figure 10] This figure shows a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. [Figure 11] This figure shows a timeline of a situational awareness surgical procedure according to one aspect of the present disclosure. [Figure 12A] This figure shows a surgical display acquired from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure, according to one aspect of the present disclosure. [Figure 12B] This figure shows an augmented reality image, including a surgical display, along with a virtual secondary view overlaid thereon, according to one aspect of the present disclosure. [Figure 13A] This figure shows a surgical display acquired from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure in which a portion of the patient's gastric fundus is removed, according to one aspect of the present disclosure. [Figure 13B]This figure shows an augmented reality image including a surgical display along with a predicted or recommended placement of staples, according to one aspect of the present disclosure. [Figure 14] This flowchart illustrates a method, according to one aspect of the present disclosure, in which an interactive surgical system can receive surgical procedure information and suggest the next procedure step. [Figure 15A] This figure shows a surgical display acquired from a surgical imaging device during a laparoscopic procedure, according to one aspect of the present disclosure. [Figure 15B] This figure shows an augmented reality image, including a surgical display, along with an augmented reality virtual object that presents the contour of the ureter beneath the tissue to be resected, according to one aspect of the present disclosure. [Figure 16] This figure shows various aspects associated with a surgical procedure that can be tracked by an interactive surgical system and analyzed to develop an optimization strategy, according to one aspect of the present disclosure. [Figure 17] This figure shows an embodiment of an operating room that may be modeled for tracking purposes, according to one aspect of the present disclosure. [Figure 18] This figure shows an example of a virtual object warning of fatigue in a surgical member, according to one aspect of the present disclosure. [Figure 19A] This figure shows several exemplary depictions of an optimized operating room according to one aspect of the present disclosure. [Figure 19B] This figure shows several exemplary depictions of an optimized operating room according to one aspect of the present disclosure. [Figure 19C] This figure shows several exemplary depictions of an optimized operating room according to one aspect of the present disclosure.

[0007] Throughout the drawings, corresponding reference numerals indicate corresponding parts. The examples described herein illustrate various disclosed embodiments in one form, and such examples should not be construed as limiting the scope. [Modes for carrying out the invention]

[0008] The applicant of the present application owns the following co-pending U.S. patent applications, the entire disclosures of each of which are incorporated herein by reference. · U.S. patent application titled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP1 / 210120-1M, · U.S. patent application titled "UTILIZATION OF SURGICAL DATA VALUES AND SITUATIONAL AWARENESS TO CONTROL THE OVERLAY IN SURGICAL FIELD VIEW"; Attorney Docket No. END9352USNP2 / 210120-2, · U.S. patent application titled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW"; Attorney Docket No. END9352USNP3 / 210120-3, · U.S. patent application titled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW"; Attorney Docket No. END9352USNP4 / 210120-4, · U.S. patent application titled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Attorney Docket No. END9352USNP5 / 210120-5, · U.S. patent application titled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Attorney Docket No. END9352USNP6 / 210120-6, · U.S. patent application titled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; Attorney Docket No. END9352USNP8 / 210120-8, · U.S. patent application titled "COOPERATIVE OVERLAYS OF INTERACTING INSTRUMENTS WHICH RESULT IN BOTH OVERLAYS BEING EFFECTED"; Attorney docket number END9352USNP9 / 210120-9, · U.S. patent application titled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Attorney docket number END9352USNP10 / 210120-10, · U.S. patent application titled "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN"; Attorney docket number END9352USNP11 / 210120-11, · U.S. patent application titled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Attorney docket number END9352USNP12 / 210120-12, · U.S. patent application titled "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION"; Attorney docket number END9352USNP13 / 210120-13, · U.S. patent application titled "COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED INFORMATION"; Attorney docket number END9352USNP14 / 210120-14, • U.S. Patent Application entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Agent reference number END9352USNP15 / 210120-15 • U.S. Patent Application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS"; Agent Reference Number END9352USNP16 / 210120-16, and, U.S. Patent Application entitled "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS"; Agent reference number END9352USNP17 / 210120-17.

[0009] The applicant of this application owns the following U.S. patent applications, the entirety of which is incorporated herein by reference: U.S. Patent Application No. 16 / 209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS" (currently U.S. Patent Publication No. 2019-0200981(A1)), U.S. Patent Application No. 16 / 209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (currently U.S. Patent Publication No. 2019-0201046(A1)).

[0010] During a surgical procedure, a group of medical professionals (surgical team) work together in the surgical room or operating room to provide interventional medical care to the patient. Depending on the surgical procedure, each member of the surgical team is assigned one or more specific roles or functions designed to be performed together and to provide medical care to the patient and achieve a favorable outcome. Each member of the surgical team may also be assigned to work with one or more components of medical devices used during the procedure. The steps and processes associated with a surgical procedure can be complex and take a considerable amount of time to perform. Furthermore, all actions of the surgical team members must be coordinated for a successful outcome. This coordination of efforts requires the proper sharing of information throughout the surgical procedure. It can be understood that the total amount of information and data, which can change significantly over time, is enormous. Therefore, sharing all potential surgical information (such as patient vital sign data, anesthesia data, and data related to the operation of each intelligent surgical device) could overwhelm members of the surgical team with false information. Surgical team members need to remain informed of data related to their specific roles and functions throughout the surgical procedure without receiving additional data unrelated to their functions.

[0011] Disclosed herein are methods and systems for distributing data to each member of a surgical team, based on their specific roles or functions, throughout the course of a surgical procedure, using an augmented reality-based surgical system.

[0012] Before describing in detail the various embodiments of surgical devices and generators, it should be noted that the illustrative embodiments are not limited in their application or use to the details of the structure and arrangement of the components illustrated in the accompanying drawings and descriptions. The illustrative embodiments may be implemented or incorporated into other embodiments, variations, and modifications, and may be carried out or performed in various ways. Furthermore, unless otherwise specified, the terms and expressions used herein have been selected for the purpose of illustrating the illustrative embodiments for the convenience of the reader and are not intended to limit them. Furthermore, it should be understood that one or more embodiments, expressions of embodiments, and / or embodiments described below may be combined with any one or more other embodiments, expressions of embodiments, and / or embodiments described below.

[0013] Various embodiments relate to on-screen displays for surgical systems for various energy and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio frequency (RF)-based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices include surgical staplers combined with electrosurgical devices and / or ultrasonic devices. Embodiments of ultrasonic surgical devices may be configured, for example, to transversely incise and / or coagulate tissue during surgical procedures. Embodiments of electrosurgical devices may be configured, for example, to transversely incise, coagulate, seal, weld and / or dry tissue during surgical procedures. Embodiments of surgical stapler devices may be configured to transversely incise and staple tissue during surgical procedures, and in some embodiments, surgical stapler devices may be configured to deliver RF energy to tissue during surgical procedures. Electrosurgical devices are configured to deliver therapeutic and / or non-therapeutic RF energy to tissue. Elements of surgical staplers, electrosurgical devices, and ultrasound devices can be used in combination within a single surgical instrument.

[0014] In various embodiments, the Disclosure provides the OR team with on-screen displays of real-time information during surgical procedures. According to various embodiments of the Disclosure, many novel and unique on-screen displays are provided for displaying various visual information feedback to the OR team on screen. According to the Disclosure, the visual information may include one or more of various visual media, with or without sound. Generally, the visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. The visual information may be communicated on any number of display options, such as, for example, a primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, etc.

[0015] In various embodiments, this disclosure provides a list of many potential options for communicating visual information to an OR team in real time without overwhelming the OR team with too much visual information. For example, in various embodiments, this disclosure provides on-screen displays of visual information that enable a surgeon, or other member of the OR team, to selectively activate on-screen displays, such as icons surrounding screen options, to manage the rich visual information. One or a combination of factors may be used to determine the active display, and these may include, among other things, the energy-based (e.g., electrosurgery, ultrasound) or machine-based (e.g., stapler) surgical device in use, the estimated risk associated with a given display, the surgeon's level of experience, and the surgeon's choice. In other embodiments, the visual information may include rich data overlaid or superimposed on the surgical field to manage the visual information. In various embodiments described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. Visual information data thus communicated can provide additional useful visual information to the OR team in a more concise and understandable way, in contrast to static icons.

[0016] In various embodiments, the Disclosure provides techniques for selectively activating on-screen displays, such as icons surrounding a screen, to manage visual information during a surgical procedure. In other embodiments, the Disclosure provides techniques for determining an active display using one or a combination of factors. In various embodiments, the techniques provided by the Disclosure may include, among other things, selecting an energy-based or machine-based surgical device to be used as the active display, estimating the risks associated with a given display, and utilizing the experience level of the surgeon or OR team making the selection.

[0017] In other embodiments, the techniques described herein may include overlaying or superimposing rich data onto the surgical field for the purpose of managing visual information. Several display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Graphical overlays may take the form of transparent graphics, translucent graphics, or opaque graphics, or combinations of transparent, translucent, and opaque elements or effects. Furthermore, overlays may be positioned on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value. Graphical overlays are rendered on an active display monitor to quickly and efficiently communicate critical information to the OR team.

[0018] In other embodiments, the technology provided by the Disclosure may include superimposing images that require video analysis and tracking in order to appropriately overlay visual information data. In other embodiments, the technology provided by the Disclosure may include communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easily understandable manner. In other embodiments, the visual overlay may be used in combination with auditory and / or somatosensory overlays, e.g., thermal, chemical, and mechanical devices, and combinations thereof.

[0019] The following description generally pertains to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information onto the real-world images of the surgical field, instruments, and / or other objects appearing in the surgical field. The images may be streamed in real time or they may be still images. Augmented reality is a technique for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid on a real environment. The real environment may include the surgical field. Virtual objects overlaid on a real environment may be represented at fixed or set positions relative to one or more aspects of the real environment. In non-limiting examples, if a real-world object moves out of the field of view of the real environment, the virtual object fixed to the real-world object also moves out of the field of view of augmented reality.

[0020] Some of the display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value.

[0021] As described herein, AR is an extended version of the real physical world achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear real, making the user feel immersed in them. This environment is perceived through a device known as a virtual reality headset or helmet. While both mixed reality (MR) and AR are considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. AR often adds digital elements to a live view by using a camera, while an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

[0022] In an AR environment, one or more computer-generated virtual objects may be displayed alongside one or more real-world (i.e., so-called "real-world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlay virtual objects. Such virtual objects can provide supplementary information about the environment or, in general, enhance the user's perception and engagement with the environment. Conversely, real-time images or videos of the surrounding environment can, in addition or alternatively, enhance the user's engagement with the virtual objects displayed on the display.

[0023] Apparatus, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during surgical procedures. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and / or cameras that provide images during incisional surgical procedures. Images may be streamed in real time or still images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing images of a real-world surgical environment by overlaying representations of virtual objects or data and / or real objects onto the real-world surgical environment. The augmented reality experience may be viewed on a display and / or AR device that allows the user to view virtual objects overlaid on the real-world surgical environment. The display may be located in the operating room or located away from the operating room. The AR device is worn on the head of a surgeon or other operating room personnel and typically includes two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through the two transparent or translucent display lenses so that aspects of the real environment are visible, while projecting light to make virtual objects visible to the user of the AR device.

[0024] Two or more displays and AR devices may be used in conjunction with a first display or AR device that controls one or more additional displays or AR devices in a system having defined roles. For example, when activating a display or AR device, the user may select a role (e.g., a surgeon, surgical assistant, nurse, etc. during a surgical procedure), and the display or AR device may display information related to that role. For example, a surgical assistant may have the display show virtual representations of instruments that the surgeon needs to use for the next step in the surgical procedure. The surgeon's focus on the current step may differ from the information displayed by the surgical assistant.

[0025] While many known on-screen displays and alerts exist, this disclosure provides many novel and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment may be provided to the operating room (OR) team, including, but not limited to, personnel within the OR, such as the surgical surgeon, surgical assistants, scrub wearers, anesthesiologists, and circulating nurses. The virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical staplers, tablets, augmented reality glasses, and other devices.

[0026] Figure 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 connected to remote storage 5. Each surgical system 2 comprises at least one surgical hub 6 that communicates with the cloud 4. For example, a surgical system 2 may comprise a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12, each configured to communicate with each other and / or with the hub 6. In some embodiments, a surgical system 2 may comprise M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12, where M, N, O, and P are integers of 1 or more. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during surgical procedures, as described herein.

[0027] Figure 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. A robotic system 10 is used as part of the surgical system 2 in the surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient-side cart 20 (surgical robot), and a surgical robot hub 22. The patient-side cart 20 allows the surgeon to operate at least one detachably connected surgical tool 17 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. Images of the surgical site during the minimally invasive procedure (e.g., still or live images streamed in real time) can be acquired by a medical imaging device 24. The patient-side cart 20 can operate the imaging device 24 to orient it. Images of the incision surgical procedure can be acquired by a medical imaging device 96. The robot hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18, or on an AR device 66 worn by the surgeon or another person in the surgical room 16.

[0028] The optical components of the imaging device 24, 96, or AR device 66 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from tissues and instruments in the surgical field.

[0029] In various embodiments, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes. In various embodiments, the imaging device 96 is configured for use in incisional (invasive) surgical procedures.

[0030] In various embodiments, the visualization system 8 comprises one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more displays. In one embodiment, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one embodiment, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range in the electromagnetic spectrum. Wavelengths are separated by filters or by instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring allows the surgical field to be repositioned after the surgical task for performing tests on the treated tissue is completed.

[0031] Figure 2 shows a primary display 19 positioned in the sterile field for the operator on the operating table 14 to see. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing opposite directions from each other. The visualization system 8, guided by the hub 6, is configured to utilize displays 7, 9, and 19 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by imaging devices 24 and 96 through the non-sterile displays 7, 9, or AR device 66, while maintaining live video of the surgical site on the primary display 19 or AR device 66. The non-sterile displays 7 and 9 can, for example, enable non-sterile operators to perform diagnostic steps related to the surgical procedure.

[0032] Figure 3 shows a hub 6 that communicates with a visualization system 8, a robotic system 10, and handheld intelligent surgical instruments 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke extraction module 26 and / or a suction / irrigation module 28. In various embodiments, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in Figure 10). Modular light sources can be adapted for use with various imaging devices. In various examples, multiple imaging devices can be positioned at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or incisional surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide the optimal view. In various embodiments, the imaging module 38 can be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during surgical procedures as described herein.

[0033] Figure 4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located in one or more operating rooms / surgery sites of a medical facility to a cloud-based system. The cloud 54 may include a remote server 63 (Figure 5) connected to a storage device 55. The modular communication hub 53 includes a network hub 57 and / or a network switch 59 that communicate with a network router 61. The modular communication hub 53 is connected to a local computer system 60 for data processing. The operating room modular devices 1a-1n may be connected to the modular communication hub 53. The network hub 57 and / or the network switch 59 are connected to the network router 61 so that devices 1a-1n can connect to the cloud 54 or the local computer system 60. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 via a wired channel or a wireless channel. The surgical data network environment 51 may be employed, as described herein, to provide an augmented reality interactive experience during a surgical procedure, in particular to provide augmented images of the surgical field to one or more remote displays 58.

[0034] Figure 5 shows a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 which may include a remote server 63. In one embodiment, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to a plurality of surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 connected to a computer system 60.

[0035] Returning to Figure 5, the modular control tower 23 is connected to an imaging module 38 connected to an endoscope 98, a generator module 27 connected to an energy device 99, a fume exhaust module 76, a suction / irrigation module 78, a communication module 13, a processor module 15, a storage array 16, and optionally smart devices / instruments 21 and sensor modules 29 connected to a display 39. Surgical site devices are connected to cloud computing resources such as a server 63, data storage 55, and a display 58 via the modular control tower 23. The robot hub 72 may also be connected to the modular control tower 23, as well as the server 63, data storage 55, and display 58. In particular, the devices / instruments 21 and the visualization system 58 may be connected to the modular control tower 23 via wired or wireless communication standards or protocols as described herein. The modular control tower 23 may be connected to a hub display 65 (e.g., a monitor, screen) to display received augmented images, including overlaid virtual objects on the real surgical world, received from the imaging module 38, device / instrument display 39, and / or other visualization systems 58. The hub display 65 may also display data received from devices connected to the modular control tower 23, along with the images and overlay images.

[0036] Figure 6 shows a surgical hub 56 including multiple modules connected to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, such as a network connectivity device, and a computer system 60 for local processing, visualization, and imaging of augmented surgical information. The modular communication hub 53 is connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53, and data associated with the modules may be transferred to the computer system 60, cloud computing resources, or both. Each of the network hubs / switches 57 / 59 within the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs / switches 57, 59 are connected to the processor 31 to provide communication connectivity to cloud computing resources and local displays 67. Communication to the cloud 54 can be done via either a wired communication channel or a wireless communication channel.

[0037] The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is connected via a system bus to a communication module 41, storage 45, memory 46, non-volatile memory 47, and an input / output interface 48. The system bus may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using various available bus architectures.

[0038] The processor 31 may include an augmented reality modeler (e.g., as shown in Figure 10) and may be implemented as a single-core or multi-core processor, such as one known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0039] System memory includes volatile and non-volatile memory. The Basic Input / Output System (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory may include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0040] The computer system 60 also includes removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0041] In various embodiments, the computer system 60 in Figure 6, the imaging module 38 in Figures 4 to 6, and / or the visualization system 58, and / or the processor module 15 may include an image processor, an image processing engine, an image processing unit (GPU), a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0042] Figure 7 shows an augmented reality system 263 that includes an intermediate signal coupler 64 located in the communication path between the imaging module 38 and the surgical hub display 67. The signal coupler 64 combines audio and / or image data received from the imaging module 38 and / or the AR device 66. The surgical hub 56 receives the combined data from the coupler 64, overlays the provided data onto the display 67, and displays the overlaid data. The imaging device 68 may be a digital video camera, and the audio device 69 may be a microphone. The signal coupler 64 may include a wireless head-up display adapter for coupling to the AR device 66 located in the communication path of the display 67 to a console that enables the surgical hub 56 to overlay data onto the display 67.

[0043] Figure 8 shows an augmented reality (AR) system including an intermediate signal coupler positioned in the communication path between the imaging module and the surgical hub display. Figure 8 shows an AR device 66 worn by the surgeon 73 to communicate data to the surgical hub 56. Peripheral information of the AR device 66 does not include active images. Rather, peripheral information includes only signals that do not have the same requirements for device settings or refresh rate. The interaction may extend the surgeon 73's information based on links with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures and, for example, ask whether an instrument is touching a nerve, blood vessel, or adhesion. The AR device 66 may include processing in the surgical hub 56 used to provide preoperative scan data, optical views, tissue examination characteristics acquired throughout the procedure, and / or answers. The surgeon 73 may write notes on the AR device 66 so that they are stored in the hub storage 45 along with patient data for later use in reporting or follow-up.

[0044] The AR device 66, worn by the surgeon 73, links to the surgical hub 56 using auditory and visual information to avoid the need for overlays, allowing for customization of information displayed around the view. The AR device 66 provides signals from devices (e.g., instruments) and responds to queries regarding location information linked with video to identify device settings or quadrants or locations. The AR device 66 has voice control and voice feedback from the AR device 66. The AR device 66 can interact with other systems in the operating room and can have available feedback and interaction wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture start commands and queries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities, including voice, visual, or haptic touch.

[0045] Figure 9 shows a surgeon 73 and a patient 74 wearing AR devices 66, and may include a camera 96 ​​in the operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects overlaid on a real-time image of the surgical field to the surgeon 73 through an augmented reality display 89 or through a hub-connected display 67. The real-time image may include parts of surgical instruments 77. The virtual objects may not be visible to others in the operating room 75 (e.g., surgical assistants or nurses), but they may also wear AR devices 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects, or may be able to see the virtual objects in augmented reality shared with the surgeon 73, or may be able to see modified versions of the virtual objects (e.g., according to customization specific to the surgeon 73), or may see different virtual objects.

[0046] Virtual objects and / or data may be configured to appear on a portion of the surgical instrument 77 or within the surgical field captured by the imaging module 38, the imaging device 68 during minimally invasive surgical procedures, and / or the camera 96 ​​during incisional surgical procedures. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides live images of the surgical area during minimally invasive surgical procedures. The AR system may present virtual objects fixed to real objects regardless of the viewpoint of one or more viewers of the AR system (e.g., the surgeon 73). For example, virtual objects may be visible to viewers of the AR system inside the operating room 75, but not visible to viewers of the AR system outside the operating room 75. Virtual objects may be displayed to viewers outside the operating room 75 when a viewer enters the operating room 75. Augmented images may be displayed on the surgical hub display 67 or the augmented reality display 89.

[0047] The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each user's eye). The screens may allow light to pass through them so that aspects of the real environment are visible while virtual objects are being displayed. Virtual objects may become visible to the surgeon 73 by projecting light. Virtual objects may appear to have some degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

[0048] An AR system may be visible to one or more viewers and may include differences between views available to one or more viewers, while maintaining some common aspects between views. For example, a heads-up display may change between two views, but virtual objects and / or data may be fixed to real objects or areas in both views. Aspects such as the color, lighting, or other changes of an object may occur between views without changing the fixed position of at least one virtual object.

[0049] Users can view virtual objects and / or data presented within the AR system as opaque or with a certain level of transparency. For example, a user can interact with a virtual object by moving it from a first position to a second position. For instance, a user may move an object with their own hand. This may be done virtually in the AR system by determining that the hand has moved to a position adjacent to or adjacent to the object (using one or more cameras, which may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controlled to move) and moving the object accordingly. The virtual form may include a virtual representation of a real-world object, or it may include visual effects such as lighting effects. The AR system may include rules to govern the behavior of the virtual object, such as exposing the virtual object to gravity or friction, or it may include other predefined rules that negate real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device 66 may include a camera 79 (which should not be confused with a separate camera 96). The AR device camera 79 or camera 96 ​​may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. The AR device 66 may project virtual items onto a representation of the real environment that the user can see.

[0050] The AR device 66 may be used, for example, in an operating room 75 during a surgical procedure performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects, such as virtual objects during the surgical procedure, to extend the surgeon's vision. The surgeon 73 may view the virtual objects using the AR device 66, a remote controller for the AR device 66, or interact with the virtual objects by using his hands to “interact” with the virtual objects or gestures recognized by the camera 79 of the AR device 66, for example. The virtual objects can extend surgical tools, such as surgical instruments 77. For example, the virtual object may appear to be connected to the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66), or to remain at a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide the surgical instrument 77 and may appear to be fixed to the patient 74. In certain examples, the virtual object may react to the movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

[0051] The augmented reality display system imaging device 38 captures real images of the surgical area during the surgical procedure. The augmented reality displays 89 and 67 present an overlay of the operating modes of the surgical instrument 77 onto the real images of the surgical area. The surgical instrument 77 includes a communication circuit 231 for communicating operating mode and functional data from the surgical instrument 77 to the AR device 66 via a communication circuit 233 on the AR device 66. The surgical instrument 77 and the AR device 66 are shown in RF wireless communication between circuits 231 and 233, as indicated by arrows B and C, but other communication technologies (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operating modes of the surgical instrument 77 that are actively visualized. The overlay combines the modes of tissue interaction in the surgical area with functional data from the surgical instrument 77. The processor portion of the AR device 66 is configured to receive operating mode and functional data from the surgical instrument 77, determine overlays related to the operation of the surgical instrument 77, and combine the tissue characteristics within the surgical area with the functional data from the surgical instrument 77. The augmented images display alerts regarding device performance considerations, non-conformity use, and incomplete capture. Non-conformity use includes out-of-range tissue conditions and tissue improperly balanced within the jaws of the end effector. Additional augmented images provide indications of incidental events, including indications of tissue tension and foreign body detection. Other augmented images display device status overlays and instrument indications.

[0052] Figure 10 shows a system 83 for augmenting images of the surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes one aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, memory 87, an AR display 89, and a camera 79. The AR device 66 may also include a sensor 90, a speaker 91, and / or a haptic controller 92. The database 93 may include image storage 94 or preoperative planning storage 95.

[0053] The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive images of instruments in the surgical field from a camera 79 or sensor 90, etc., and create an augmented reality environment that fits within the displayed image of the surgical field. In another example, physical objects and / or data may be overlaid on the surgical field and / or surgical instrument images, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of virtual objects and / or data within the augmented reality environment. For example, the augmented reality modeler 86 may use or detect instruments at the patient's surgical site and present virtual objects and / or data on the surgical instruments, and / or images of the surgical site in the surgical field captured by the camera 79. The AR display 89 may display the AR environment overlaid on the real environment. The display 89 can use the AR device 66, which is located in a fixed position, etc., within the AR environment, to show virtual objects and / or data.

[0054] The AR device 66 may include sensors 90 such as infrared sensors. The camera 79 or sensor 90 may be used to detect movements such as gestures by a surgeon or other user, which may be interpreted by the processor 85 as attempted or intended interactions by the user with a virtual target. The processor 85 can identify objects in the real environment by processing information received using the camera 79, for example. In other embodiments, sensor 90 may be a tactile sensor, audible sensor, chemical sensor, or thermal sensor to generate corresponding signals that can be combined with various data feeds to create an augmented environment. Sensor 90 may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye-tracking sensors, and voice command recognition capabilities.

[0055] The AR display 89 may, for example, during a surgical procedure, allow the surgical field to be viewed through the AR display 89, while presenting virtual features within the surgical field that correspond to physical features hidden by the patient's anatomical features. The virtual features may have a virtual position or orientation that corresponds to a first physical position or orientation of the physical features. In one example, the virtual position or orientation of the virtual features may include an offset from the first physical position or orientation of the physical features. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical features, and so on.

[0056] In one example, the AR device 66 may be an individual AR device. In one embodiment, the AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft in Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometer, gyroscope, magnetometer), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition capabilities. It provides a high-resolution, improved field of view by using mirrors to orient waveguides in front of the wearer's eyes. The image can be magnified by changing the angle of the mirrors. It also provides eye tracking to recognize the user and adjust the lens width for a particular user.

[0057] In another example, AR device 66 could be the Snapchat Spectacles 3 AR device. This AR device offers the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras for capturing 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to construct a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synchronized to an external display device.

[0058] In yet another example, AR device 66 could be Google's Glass 2 AR device. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

[0059] In another example, AR device 66 could be an Echo Frames AR device from Amazon. This AR device does not have a camera / display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

[0060] In yet another example, AR device 66 could be the Focals AR device by North (Google). This AR device provides a notification pusher / smartwatch analog, inertial measurement, screen overlays for information (weather, calendar, messages), and voice control (Alexa) integration. This AR device also provides basic head-up display functionality.

[0061] In another example, AR device 66 could be an Nreal AR device. This AR device includes spatial sound, two ambient cameras, a photographic camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and proximity sensor functions. Nebula projects application information onto the lens.

[0062] In various other examples, the AR device 66 may be any one of the following commercially available AR devices, namely Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, or EyeTap, which create light collinear with the ambient light directly onto the retina. A beam splitter makes the same visible light available to a computer, for example, to process and overlay information. The AR visualization system may include a HUD, contact lenses, glasses, virtual reality (VR) headset, virtual retinal display, intraoperative display, and / or smart contact lenses (bionic lenses).

[0063] The multi-user interface for the AR device 66 includes a virtual retinal display such as a raster display that draws directly onto the retina rather than on a screen in front of the eyes, a smart TV, a smartphone, and / or a spatial display such as the Sony Spatial Display System.

[0064] Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, the Apple Polycam app and Ubiquity 6 (Mirrorworld using the Display.land app), which allow users to scan and acquire 3D images of the real world (to create 3D models). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, and Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, and Magic Leap.

[0065] Situational awareness is the ability of several embodiments of a surgical system to determine or infer information related to a surgical procedure from data received from a database and / or instruments. This information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. Contextual information relating to a surgical procedure can improve a surgical system, for example, by controlling modular devices connected to it (e.g., robotic arms and / or robotic surgical tools) and providing contextualized information or suggestions to the surgeon during the course of the surgical procedure.

[0066] Figure 11 shows a timeline of a situation-aware surgical procedure. Figure 11 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each step of the surgical procedure. Timeline 5200 shows typical steps that nurses, surgeons, and other healthcare workers might take during a lung segmentectomy procedure, which begins with setting up the operating room and ends with transferring the patient to the postoperative recovery room. Throughout the course of the surgical procedure, the situation-aware surgical hub 5104 receives data from data source 5126, including data generated each time healthcare workers use the modular device 5102 paired with the surgical hub 5104. By receiving this data from the paired modular device 5102 and other data sources 5126, the surgical hub 5104 can continuously derive estimations (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub 5104 can, for example, record data relating to a procedure to generate a report, verify the steps being taken by a healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as described above.

[0067] In the first step 5202, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery.

[0068] In the second 5204, staff scan incoming medical supplies for a procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies used in various types of procedures to confirm that the mixture of supplies corresponds to a thoracic procedure. Furthermore, the surgical hub 5104 can also determine that the procedure is not a wedge resection (because the incoming supplies either do not contain specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection).

[0069] In the third 5206, a healthcare worker scans the patient band via a scanner 5128 that is communicably connected to a surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

[0070] In the fourth part of 5208, a medical professional turns on the assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they include a fume exhauster, an air inlet, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104 located within a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on the combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team. Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing.

[0071] In step 5210, the staff attaches the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 confirms that the patient is in the operating room.

[0072] In step 6, 5212, medical personnel induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from modular devices 5102 and / or patient monitoring devices 5124, including, for example, EKG data, blood pressure data, ventilator data or a combination thereof. Once step 6, 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion commences.

[0073] In section 7 of 5214, the lung of the patient being operated on collapses (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has begun and determine that collapsing the lung is the first surgical step in this particular procedure.

[0074] In step 8, 5216, a medical imaging device 5108 (e.g., a scope) is inserted, and video footage from the medical imaging device is initiated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or real-time live-streaming video) through its connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resection has not been taken into consideration by the surgical hub 5104 based on the data received in step 2 of the procedure, 5204). Using data from the medical imaging device 124 (Figure 2), contextual information about the type of procedure being performed can be determined in various ways, for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number of medical imaging devices being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used.

[0075] For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. The situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0076] In step 9, 5218, the surgical team initiates the incision step of the procedure. The surgical hub 5104 receives data from the RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the procedure steps described above) corresponds to the incision step.

[0077] In the tenth step 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 receives data from the surgical stapling and cutting instruments indicating that instruments are being fired, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps in the read-out process.

[0078] In the eleventh step 5222, the segmental resection portion of the procedure is performed. The surgical hub 5104 estimates that the surgeon has transversely incised parenchymal tissue, based on data from surgical instruments, including data from a staple cartridge. Cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Cartridge data may indicate the type of tissue being stapled and / or transversely incised, for different types of staples used for different types of tissue. The type of staples fired is used for parenchymal tissue or other similar tissue types, and the surgical hub 5104 can estimate that a segmental resection procedure has been performed.

[0079] Next, in the twelfth step 5224, the nodule incision step is performed. Based on the data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby enabling the surgical hub 5104 to make this inference. Note that the surgeon will periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once the twelfth step 5224 is completed, the incision is closed and the postoperative portion of the procedure begins.

[0080] In the 13th step 5226, the patient is de-anesthetized. The surgical hub 5104 can estimate that the patient is waking up from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase).

[0081] Finally, in the 14th step 5228, the healthcare worker removes the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 can infer that the patient has been transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. Based on the data received from various data sources 5126 that are communicably connected to the surgical hub 5104, the surgical hub 5104 can determine or infer when each step of a given surgical procedure is occurring.

[0082] As shown in the first step 5202 of the timeline 5200 depicted in Figure 11, in addition to using patient data from the EMR database(s) to estimate the type of surgical procedure being performed, the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0083] Augmented reality display for invisible treatment steps It can be understood that a computer-implemented interactive surgical system may include one or more surgical systems and a cloud-based system. The cloud-based system may include a remote server connected to a storage device. Each surgical system has at least one surgical hub that communicates with the cloud. For example, a surgical system may include a visualization system, a robotic system, and one or more handheld intelligent surgical instruments, each configured to communicate with each other and / or with the hub. The surgical hub can dynamically determine which devices are in use, their locations relative to each other, and their locations relative to important structures and anatomical structures identified by the system. Based on the locations of these devices, the patient's anatomical structure, and the procedure steps in the operating room, an augmented reality display may be updated to depict one or more auxiliary augmented reality views. Such auxiliary augmented reality views may consist of views within the surgical field that the surgeon cannot see in the primary field of view. In one embodiment, such auxiliary augmented reality views may depict anatomical structures that are obscured in the primary field of view by other tissues. In another embodiment, such an augmented reality view can depict a view of a handheld intelligent surgical instrument from a secondary viewpoint (e.g., an underside view of a handheld intelligent surgical instrument).

[0084] Computer-implemented interactive surgical systems can be recognized as continuously acquiring device position and usage data during procedures. Interactive surgical systems can also continuously receive visual tracking or imaging information from various intelligent surgical instruments or other devices regarding the patient's anatomical structure. The surgical system may also retain imaging information throughout the surgical procedure. The surgical system may also retain imaging information of the patient's anatomical structure from previous surgical procedures through connectivity to a cloud-based system.

[0085] A computer-implemented interactive surgical system may determine the state of an intelligent surgical instrument based on the movement of the device while it is being used. Such movement data can be obtained from the intelligent device itself, for example, based on a triaxial accelerometer placed within the device. Alternatively, the movement data may be obtained from a visualization device capable of optically tracking the movement of the surgical instrument. The interactive surgical system may also include an anatomical image recognition algorithm configured to receive imaging data of anatomical structures and determine their properties and location. The combination of the movement of the surgical device and the determination of the anatomical structures around the surgical device may be used by the interactive surgical system to identify the current step of a surgical procedure.

[0086] In some embodiments, an interactive surgical system may use imaging data of a surgical device acquired from a visualization device to determine whether the surgical device is the appropriate device for the current step in a surgical procedure. The augmented reality device may provide a virtual object as a warning, such as an icon or text box, to overlay on the image of the surgical device in the augmented reality display, in order to provide the surgical device user with a warning that the device is not correct for that procedure.

[0087] In steps where the surgeon cannot see a specific portion of the end effector of an intelligent surgical device, the augmented reality display may include a secondary view generated to show the surgeon the location of the device that cannot be seen in the current field of view. As an example, Figure 12A depicts a surgical display 11000 acquired from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure in which a portion of the patient's stomach fundus is removed. Figure 12A depicts the patient's stomach 11002 while the surgeon is stapling the inner edges of the stomach 11002 together using a stapler 11004 and resecting the remaining stomach portion 11003. As can be observed, the surgical display 11000 depicts the upper surface 11006 of the end effector of the stapler 11004. The surgeon may want to see the bottom side of the stapler 11004 before firing the staples to ensure that both edges of the resected stomach are sealed together. A surgeon may not want to rotate the stapler 11004 while it is clamped to the stomach 11002 in order to view the rear side of the device before firing. Such rotation of the stapler 11004 may result in pulling on the tissue in a way that could impair its stapling function. Figure 12B depicts an augmented reality image 11012 that includes a surgical display 11000 with a virtual secondary view 11010 overlaid on it. The virtual secondary view 11010 may include an augmented reality depiction of a lateral view 11014 of the stapler. The augmented reality depiction of the lateral view 11014 of the stapler may depict a lateral view 11016 of a portion of the patient's stomach grasped by the mandible 11018 of the stapler. In this way, the virtual secondary view 11010 can allow the surgeon to visualize the underside of the stapler and view the underside without having to excessively manipulate the tissue.

[0088] In some embodiments, the augmented reality image 11012 can display information in addition to imaging views unavailable to the surgeon (such as a side view of the stapler 11014 and a side view of a portion of the patient's stomach 11016). For example, the virtual secondary view 11010 may also include visual indicators 11020 regarding the procedure or tissue condition. For example, a warning 11021 indicating the tissue condition may be depicted. In another example, a device status indicator 11022 may indicate the current operating status of the stapler.

[0089] In one embodiment, the virtual secondary view 11010 may be created using a predictive model of the patient based on previous anatomical images of the patient or images of similar anatomical parts of other patients undergoing the same surgical procedure. In another embodiment, the virtual secondary view 11010 may be created from real-time images acquired from a secondary camera used during the procedure. In one example, the surgeon may request the virtual secondary view 11010 from the interactive surgical system through the use of gestures or verbal commands. The interactive surgical system may alert the surgeon to adjust the position of the secondary camera so that an auxiliary view can be created.

[0090] In another embodiment, a virtual secondary view 11010 can be used to identify and display important anatomical structures at all times during a surgical procedure. For example, such a persistent secondary view can be used to maintain an image of a tumor located on the organ of interest on the display device throughout the surgical procedure. Such a persistent display can allow the surgeon to switch between overlaying this view on the image of the current procedure and having it as a secondary view on the side of the display.

[0091] Predictive analytics and AI learning through treatment - Active device display power It can be understood that a computer-implemented interactive surgical system may include one or more surgical systems and a cloud-based system. The cloud-based system may include a remote server connected to a storage device. Each surgical system has at least one surgical hub that communicates with the cloud. For example, a surgical system may include a visualization system, a robotic system, and one or more handheld intelligent surgical instruments, each configured to communicate with each other and / or with the hub. The surgical hub can dynamically determine which devices are in use, their locations relative to each other, and their locations relative to important structures and anatomical structures identified by the system. In addition, a computer-implemented interactive surgical system and / or a cloud-based system may include an artificial intelligence ("AI") system configured to monitor data drawn from previous cases of the same procedure type and import data from the current case. Cloud-based data specific to the surgeon performing surgery in a particular surgical case (and / or all completed surgical cases of this type) may enable the AI ​​system to recognize the current procedure step and use this information to predict the next steps in the procedure. Using this prediction, the augmented reality display may be updated to present the predicted next action that should be performed and / or the predicted outcome based on previous cases.

[0092] A computer-implemented interactive surgical system may determine the state of an intelligent surgical instrument based on the movement of the device while it is being used. Such movement data can be obtained from the intelligent device itself, for example, based on a triaxial accelerometer placed within the device. Alternatively, the movement data may be obtained from a visualization device capable of optically tracking the movement of the surgical instrument. The interactive surgical system or AI system may also include an anatomical image recognition algorithm configured to receive imaging data of anatomical structures and determine their properties and location. The combination of the movement of the surgical device and the determination of the anatomical structures around the surgical device may be used by the interactive surgical system to identify the current step of a surgical procedure.

[0093] A computer-implemented interactive surgical system may be recognized as continuously acquiring device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices regarding the patient's anatomical structure. The surgical system may also retain imaging information throughout the surgical procedure. The surgical system may also retain imaging information of the patient's anatomical structure from previous surgical procedures, or imaging information from different patient anatomical structures from related surgical procedures, through a connection to a cloud-based system.

[0094] In one embodiment, data may be sent to a cloud data source configured to store in memory all previous procedure data from additional hub-connected cases. The data may be mined and analyzed to predict the most likely next steps to be taken by the surgeon. Non-limiting examples of predictive modeling may include using one or more of classification models, regression models, and Markov chain models. Previous procedure data may include imaging data and data acquired from specific devices while they are used in the procedure. Device-dependent data may include, for example, power level, timing parameters, staple type, device position, and orientation data, along with other operating parameters. The surgical cases to be analyzed may include any number of related or identical procedures. In some examples, only related cases completed by a specific surgeon performing the procedure may be analyzed. In one embodiment, the surgeon performing the current procedure may have the option to select which previous cases should be analyzed as relevant to the case at hand in order to make predictive recommendations.

[0095] Using surgeon-specified predictions, tracked position and orientation of surgical devices in use, and the patient's anatomical structure, the augmented reality display may be updated to show predictions for the next surgical action (e.g., predicted position of the stapler for the next stapling operation). It can be recognized that the augmented reality display may be shown on any display device inside or outside the operating room. In some embodiments, the augmented reality display may be displayed on a primary or main display inside the operating room. Alternatively, the augmented reality display may be displayed on one or more alternative displays, such as a tablet device, a secondary monitor, or a display device associated with a specific intelligent surgical device, such as a device generator display. This prediction may be accompanied by additional device-specific recommendations, such as a predicted staple reload size based on the patient's anatomical structure, or it may suggest the use of buttress material to reduce leakage based on observations from previous stapling operations, the patient's disease state, etc.

[0096] Other augmented reality displays may include recommendations related to the use of additional surgical devices that can be used to complete the procedure with improved results. By tracking procedure steps during the surgical procedure in progress and comparing them with previously acquired data stored in a cloud system, the intelligent surgical system may also coordinate communication prioritization between intelligent surgical devices in a hub network. Thus, based on the surgical history, a second intelligent surgical device that will be needed after the use of a first surgical device may have its communication stream prioritized in anticipation of its use. For example, in a gastric sleeve procedure, after all stapler firing is complete, the communication stream from the needle holder may be prioritized over other devices.

[0097] As an example, Figure 13A depicts a surgical display 11000 acquired from a surgical imaging device during a laparoscopic sleeve gastrectomy procedure in which a portion of the patient's gastric fundus is removed, similar to the depiction in Figure 12A. Figure 13A depicts the patient's stomach 11002 while the surgeon is using a stapler 11004 to staple the inner edge of the stomach 11002 together and cutting the remaining stomach portion 11003. As can be observed, the surgical display 11000 depicts the upper surface 11006 of the end effector of the stapler 11004 at a specific location on the stomach 11002. The surgeon may be uncertain regarding how or where to position the stapler 11004 for the subsequent stapling and cutting operations.

[0098] Figure 13B depicts an augmented reality image 11030 including a surgical display 11000 along with a predicted or recommended placement of a stapler 11032. In some embodiments, the augmented reality image 11030 may display supplementary information 11034 in addition to the predicted or recommended placement of the stapler 11032. For example, the supplementary information 11034 may include recommendations 11036 regarding changes to device operating parameters, such as the type of stapler used by the stapler. The recommendations may also include statistical data regarding the success rate of the new operating parameters in similar surgeries. In another example, the supplementary information 11034 may also include one or more warnings 11038 regarding the condition of the tissue being manipulated by the surgical device. The warnings 11038 may include recommended repair steps that can be used to address the problem. For example, the warning 11038 may indicate that the staple line is stressing the tissue, which may result in rupture or incomplete healing at the staple line. A recommendation may be given suggesting the use of buttress material to help seal the tissue.

[0099] Figure 14 is a flowchart 11500 illustrating how an interactive surgical system may receive surgical procedure information and suggest the next procedure step. In the first step of process 11510, the surgeon completes surgical step A. Data related to surgical step A may be sent from the communication hub to a cloud-based data source (11512). An artificial intelligence engine in the cloud-based data source can predict the next steps in the surgical procedure (11514) and transmit that data to the communication hub. The communication hub can then predict the surgical field and location within the surgical device that may be used by displaying one or more virtual objects on an augmented reality display (11516). The communication hub can then communicate with the cloud data source (11518) to determine the best surgical outcome for the predicted steps, locations, and devices based on parameters associated with the patient. These parameters may be compared with similar patient statistics, surgical information, and surgical device characteristics in a cloud-based database. The hub may communicate parameters related to use, orientation, location, and device parameters to the surgeon as a virtual object displayed on the surgeon's associated augmented reality display device (11520). The surgeon may then complete the recommended next surgical step (11522). This method may thus be continued for each subsequent setting in the surgical procedure.

[0100] Related equipment, user devices, and interaction hub / network sensing A computer-implemented interactive surgical system may be recognized as continuously acquiring device position and usage data during a procedure. The interactive surgical system may also continuously receive visual tracking or imaging information of various intelligent surgical instruments or other devices regarding the patient's anatomical structure. The surgical system may also retain imaging information throughout the surgical procedure. The surgical system may also retain imaging information of the patient's anatomical structure from previous surgical procedures, or imaging information from different patient anatomical structures from related surgical procedures, through a connection to a cloud-based system.

[0101] An augmented reality interactive surgical system includes multiple data-connected intelligent surgical devices. The surgical system can dynamically determine which devices are in use and where they are located in space relative to each other and critical structures and anatomical structures so that they can be identified by the system. Based on the location of these devices and user preferences / locations, the system may prioritize data communication. Data communication prioritization may be enabled for devices close to critical structures and may include increased alert sensitivity around critical procedure steps and / or specific anatomical structures. In response to data communication prioritization, augmented reality displays may be rapidly updated to inform the surgeon or other members of the operating room staff of device locations, high-risk areas, and other sensitive locations.

[0102] Through spatial tracking, the interactive surgical system can adapt the augmented reality display to the procedure step, such as highlighting important structures when instruments are nearby or setting alerts when instruments are too close to each other. Thus, while augmented reality information is constantly tracked by the system, the information is only displayed when it is important to the surgeon. For example, augmented reality visualization of hidden structures is not always enabled but can be triggered by the location of the intelligent medical device. In this way, the surgeon can proceed with the surgery as usual until a high-risk area is identified or until there is a risk of unintended damage. In such situations, augmented reality visualization may be enabled, and the surgeon will be notified of any imminent problems.

[0103] For example, an interactive surgical system can recognize the location of important anatomical structures, such as the ureter, which might otherwise be invisible to the surgeon. For instance, an artificial intelligence module in a cloud system may include an anatomical model that can capture images of the surgical field and predict or estimate the location of nearby or underlying anatomical structures. While the ureter may not be easily visible, an image of the ureter could appear as an augmented reality virtual object on one or more augmented reality displays. In one option, such an augmented reality virtual object could be displayed when the system detects / predicts that the end effector of a connected device has come within a specified distance of this important structure.

[0104] In another example, the augmented reality virtual object may include a highlight superimposed on the display of the end effector of the ultrasound instrument when the temperature reaches a certain level and the instrument is in close proximity to the intestinal wall.

[0105] Alternative display options may be based on the surgeon's preference. Some options may include a persistent display of augmented reality visualization of critical structures. Alternatively, the display of augmented reality visualization of critical structures may be enabled only during certain parts of the procedure or only when the energy device is in use. These visualization options may depend on a combination of monitoring the device's position relative to the patient's anatomical structures (e.g., via scopes, scans), processing this information in an interactive surgical system, and enabling the desired augmented reality display based on the surgical context.

[0106] As an example, Figure 15A depicts a surgical display 11040 acquired from a surgical imaging device during a laparoscopic procedure. The procedure may include the use of an ultrasonic cutter 11042 and an auxiliary tissue clamp 11044. The surgeon may want to grasp a tissue piece 11046 with the tissue clamp 11044 and use the ultrasonic cutter 11042 to excise a portion of it. The surgeon may not realize that a portion of the patient's ureter is directly beneath the tissue 11046.

[0107] Figure 15B depicts an augmented reality image 11050 including a surgical display 11040, along with an augmented reality virtual object 11056 that presents the contour of the ureter beneath the tissue to be excised. In some embodiments, the augmented reality image 11050 may display auxiliary information 11057 in addition to the augmented reality virtual object 11056. For example, the auxiliary information 11057 may include a tissue-related warning 11058 indicating that the ultrasonic cutter 11052 or auxiliary tissue clamp 11054 is too close to the ureter. The auxiliary information 11057 may also include a device-related warning 11059 indicating that the ultrasonic cutter 11052 may be too close to the auxiliary tissue clamp 11054.

[0108] Augmented reality displays may be used to provide surgeons with any relevant guidance related to a procedure or the devices used in the procedure. For example, if an interactive surgical system determines, based on the surgeon's skill history, that a trainee surgeon (such as a surgical resident) lacks experience, the trainee surgeon may receive more feedback in user training mode. The amount of feedback presented may be graded based on a “training curve” related to the surgeon’s skill level and may be accelerated if it shows improvement in the surgeon’s learned skills. The feedback may be tailored to provide guidance in areas where skill improvement is needed. Personalized feedback may be based on data and images stored in a cloud system for individual surgeons, based on past performance and surgical experience, as well as recorded use of devices by the surgeon during past surgeries. Examples of surgical outcomes that may indicate a need for skill improvement may include bleeding at the surgical site, double burns of cauterized tissue, or tissue tagging.

[0109] The augmented reality display may also be used to recommend specific devices to the surgeon during a procedure. Improved or updated devices may be recommended to replace surgical devices currently in use during a procedure. Information showing how such improved devices can be used in the current surgery may be provided on the augmented reality display. The augmented reality display may also provide statistics on the outcomes of similar surgeries performed using the recommended device compared to the outcomes of surgeries using the device.

[0110] Linked users to form an AR ecosystem Computer-implemented interactive surgical systems can be recognized as continuously acquiring device position and usage data during a procedure. Interactive surgical systems can also continuously receive visual tracking or imaging information from various intelligent surgical instruments or other devices regarding the patient's anatomical structure. Augmented reality interactive surgical systems comprise multiple data-connected intelligent surgical devices and one or more display devices configured to provide members of the surgical team with information related to the surgical procedure, patient condition, and the operation of the intelligent surgical devices used throughout. The surgical system can dynamically determine which devices are being used and their spatial position relative to each other and to important structures and anatomical structures so that they can be identified by the system.

[0111] In some embodiments, each member of a surgical team may be associated with one or more display devices to provide information related to surgical procedures. Each display device may display images acquired from one or more imaging devices, along with augmented reality virtual objects overlaid on the imaging data. A display associated with any member of the surgical team may be customized to suit the functional role of that member. For example, a display for a member of the surgical team may be customized to include virtual objects associated with devices or instruments controlled by that member. An interactive surgical system can monitor instruments and devices under the control of each member of the surgical team in the operating room. The data displayed on each display device may depend on which user has control of the surgical device and the user's surgical role. The information displayed for a user may change when an instrument or device enters or leaves their control. For example, the surgical system may track instrument exchanges between surgeons or between a surgeon and a nurse. Augmented reality displays may adjust the nature, type, and / or position of virtual objects within the augmented reality display associated with the surgical team member involved. For example, a virtual object associated with the control of a surgical device may disappear from the display associated with one surgical team member if that member relinquishes control of the device. Similarly, a virtual object associated with the control of a surgical device may reappear on the display associated with a second surgical team member if that second member accepts control of the device.

[0112] In some embodiments, the intelligent surgical system can determine the original position and external configuration of the instrument being used, as well as the associated user controlling the instrument. The surgical system can also correlate externally occurring actions with internally occurring movements to verify the correct correlation between the two facets.

[0113] As disclosed above, each member of the surgical team may be associated with their own display device configured to display an augmented reality display customized to the activities and roles of the surgical team member. Such displays may include any appropriate type of display, including a primary or main operating room display, one or more auxiliary operating room displays, displays associated with one or more tablet devices, laptop displays, smartphone displays, or displays associated with individual surgical devices such as patient monitoring devices or anesthesia delivery / monitoring devices. The purpose of each of these devices is to provide information customized to the individual's functional role.

[0114] Not only can displays be customized to suit specific surgical team members, but team members may also be able to modify their own displays to point to the displays of other team members, for example, by swiping on a touch-activated screen, using hand gestures, or by verbal commands. In this way, team members may be able to "pull" a display from another team member's display device. Alternatively, some team members may have the authority to "push" their own displays onto the display devices of other members of the surgical team.

[0115] In some embodiments, each member may not have a physical display device. Instead, multiple members of the surgical team may rely on a shared or common display device. In this scenario, the customized experience can be derived from a wearable image filtering device such as glasses, a head-up display, or contact lenses. The common display device may display an image containing virtual objects associated with all members of the surgical team. The virtual objects may be color-coded or otherwise visually encoded so that each member of the surgical team can see only the virtual objects associated with them by using the wearable image filtering device. The wearable image filtering device may filter the image displayed on the common display device based on rapidly changing image color filtering, polarization filtering, or temporal filtering. Color and polarization filtering may allow the user to see only light emitted at a pre-selected wavelength or polarization state. Temporal filtering can adjust the timing of blank moments in the wearable filter to the time when a particular image is displayed on the common display device. Color-filtering contact lenses or goggles may be used for rapid prototyping and information gathering at the information level. Alternatively, a zoom feature or a UV light activation option may be incorporated into the wearable filter.

[0116] Interactive surgical systems may also manage the priority of communication among members of the surgical team and / or with a hub or cloud system. Therefore, communication arising from functions or activities deemed critical to a procedure at any given time may take precedence over, for example, communications associated with routine patient monitoring. These communications may take precedence among members of the surgical team. Thus, relevant data generated by critical devices used during specific parts of a procedure may be communicated directly to all relevant members of the surgical team to convey useful information about the devices. In a non-limiting example, a surgeon may need to know at all times whether an ultrasonic blade is hot, while the anesthesiologist only needs to know when the hot blade comes into contact with a critical anatomical structure. In this case, the anesthesiologist can be notified only when a critical anatomical structure is in contact with the hot blade, via vibration feedback associated with the temperature of the ultrasonic blade. The surgeon may receive tactile feedback, e.g., a light vibration response, from the hot blade and receive notification / light vibration when the blade approaches a critical structure overall. In another example, devices and / or procedures that result in hemostasis difficulties may take precedence. Data specific to the procedure that may be necessary to monitor hemostasis, such as the type and source of blood flow, may be shared among the surgical team.

[0117] Each member of the surgical team may have their own customized augmented reality display, although individual display options may be disabled depending on the situation. For example, during an emergency, everyone may see the same display or receive the same prioritized alerts based on the detected situation. All members of the surgical team have access to a standard, non-negotiable master setting or standard view. However, each individual can add additional settings and preferences to the standard image.

[0118] In some embodiments, communication between specific members of a surgical team may be prioritized to the extent that information is paired with specific members for direct communication. For example, the head surgeon may share what he is seeing, and his display / wearable preferences may be extended to specifically selected members of the surgical team. For instance, the head surgeon may "push" his display to a surgical resident, so that the surgical resident can see or feel the same thing as the head surgeon. A ping system with wearable devices may notify other surgical team members to switch their respective augmented reality displays. For example, a surgeon or surgical assistant may ping an anesthesiologist to switch their view / preferences to the surgeon's view / preferences. It may be understood that surgical team members who are thus pinged may decline the invitation if they have higher-priority tasks at hand.

[0119] The surgical system may initiate communication pairing with an intelligent surgical device and then associate the device's user with the device while the device is active. It is disclosed above that the intelligent surgical device may be controlled by a member of the surgical team. The intelligent surgical system may recognize the association of a person with the device after the intelligent surgical system has established communication pairing with the device itself. The system may first determine that the device is located in the surgical room. The system may then recognize that the device has been removed from its sterile packaging, thereby becoming passively selectable for use by a member of the team. Once a member of the surgical team begins handling the device, the interactive surgical system may recognize the state of the intelligent device as actively selected, and the system may therefore recognize control or association between the device and the member of the surgical team.

[0120] Device monitoring - changes in user interaction and display needs An intelligent surgical system can determine the tasks of surgical team members based on situational awareness of the intelligent surgical instrument status, the functional roles of team members, and one or more steps in the procedure. Situational awareness may be used to adapt augmented reality virtual object display information based on the task at hand.

[0121] In a non-limiting example, a surgeon can deploy a loaded stapler positioned over an organ to be transversely incised. The augmented reality display may include virtual objects displaying information related to the firing force (FTF), standby time, and detected tissue tension, as well as one or more of the cartridge color, status, and stroke location of the intelligent medical device. Once firing is complete, the surgeon can release and close the tissue and remove the device. Here, since the device has consumed a cartridge, the augmented reality display may indicate that the stapler cannot be used again until it is reloaded. The augmented reality display may use a colored virtual object as an overlay on the image of the stapler to indicate that the stapler is not available for further use until it is reloaded. When the stapler is handed to a scrubbing nurse, the virtual object representing the stapler can be transferred to the nurse's augmented reality display. Simultaneously, the virtual object representing the stapler may be removed from the surgeon's augmented reality display to indicate that control of the device has been transferred to the nurse. The nurse's augmented reality display may then show the firing status of the stapler and the steps required to reload the instrument with a new, unfired cartridge. Such virtual objects may include indications regarding the stapler buttons to press, their order, and suggestions for the next cartridge color, based on the treatment plan and the intelligent medical device's recognition of the steps it is currently in. In some embodiments, the nurse's augmented reality display may also link to other displays to show where the required cartridges are located, and even the compatibility of the cartridges loaded with the device and treatment.

[0122] In some embodiments, an interactive surgical system can track multiple aspects of an operating room procedure. In some examples, the interactive surgical system may track the use and control of multiple intelligent surgical devices used during the procedure, the location and activities of surgical team members, and the placement of instruments and patients within the scope of the surgical ROM itself. Additional traceable embodiments may include surgical access points that can be registered to patients, instrument locations, orientations, or statuses, or lost or misplaced instruments. In some additional embodiments, the interactive surgical system can identify ongoing surgical procedure steps. The surgical system may display virtual objects with background highlighting on one or more augmented reality displays to indicate critical or time-sensitive steps, or to indicate that certain steps are at a higher risk level than others.

[0123] In some embodiments, an interactive surgical system can track the skills or abilities of members of the surgical team within the operating room. In addition, the interactive surgical system can track the individual locations and functions of members of the surgical team within the operating room. It can also track the locations of personnel entering and leaving the operating room. The interactive surgical system can monitor the movements, interactions, and actions of surgical staff within the operating room to improve the layout. Embodiments related to the use of intelligent surgical devices, such as the dominant hand of a team member using such a device, or the placement of surgical devices on a table before or after use to optimize efficiency, may also be tracked.

[0124] Figure 16 depicts various aspects associated with a surgical procedure that can be tracked by an interactive surgical system and analyzed to develop optimization strategies. Patient-related data may include the patient's location, as well as pre- and post-operative scheduling. Material data may include a list and location of consumables such as gauze and wipes, as well as their management in the hospital supply chain. Location-related data may include not only the operating room but also auxiliary rooms such as storage facilities and workspaces. Controls may include rules, regulations, and laws related to the hospital's operational practices and medical, political, economic, and legal constraints on those practices. Functional data is all data related to the individual steps, processes, and sub-processes included in the surgical procedure. Flow data includes the actual sequence of procedures and processes, including models. Operational data concerns the application of various components of surgical instruments, along with devices and tools. Information includes all data, documents, and data models of the procedure acquired during the procedure. Finally, organizational data may include personnel data from the hospital, including staff identification, surgical roles, and an organizational model of the hospital itself.

[0125] Figure 17 depicts an embodiment of an operating room 11100 that may be modeled for traceability purposes. The depicted operating room 11100 may simply represent an operating room in general, and the layout may be adapted to any real-world example of an operating room. The operating room 11100 has access 11102 through which patients, surgical team members, and equipment can enter. The operating room may have an exit 11116 through which used equipment can be disposed of or recycled. There may be a central operating area 11118 including a sterile area 11108 for patients and surgeons, and an anesthesia area 11110 for anesthesiologists and nurse anesthesiologists. The central operating area 11118 may also include a location for any surgical robots involved in the surgery. Surrounding the central operating area 11118 may be a circulation area 11106 through which nurses, technicians, and other individuals can move so as not to interfere with the surgical procedures performed in the operating area 11118. A medical record entry area 11104 where clinical records can be taken may be accessible through the circulatory area 11106. Outside the operating room 11100, there may be a perimeter passageway 11112 which may include a scrub station 11114 for surgical team members to wash their hands and put on their personal protective equipment.

[0126] An interactive surgical system can track changes in the posture of surgical team members, such as swaying back and forth while standing. Such postural activity may indicate an increase in the fatigue level of the surgical team member. In response, the interactive surgical system can display virtual objects on an augmented reality display associated with the fatigued member of the surgical team. By displaying the appropriate virtual object on each augmented reality display, the interactive surgical system can alert support staff to potentially allow the current staff or surgeon to take a break. In one embodiment, the virtual object may be an indicator of the team member's fatigue state. An example of such a virtual object warning for surgical member fatigue is depicted in Figure 18.

[0127] As part of the capabilities of an interactive surgical system to track multiple aspects of an operating room procedure, the interactive surgical system may include optimization algorithms to improve performance during the surgical procedure. Such optimizations may include optimizing the correct surgical tools to be used at the correct time, such as determining that all necessary surgical tools are available and conveniently positioned before the start of the procedure. The system may also determine an optimized operating room layout for equipment, an optimized patient placement, or optimized access to equipment or operating room doors.

[0128] Once the procedure is complete, the interactive surgical system can analyze and suggest changes to the operating room flow to minimize inefficient areas. This analysis may influence the procedure planning for all future similar surgeries. This analysis can be tailored to each unique operation. The size and shape of the operating room, the number of staff, the placement of entrances and exits, the location of supplies within the operating room, and the external materials required to enter the operating room can all be analyzed by the interactive surgical system, taking into account the movements and actions of the surgical team members during the surgery, as well as the outcome of the surgery. The interactive surgical system can virtually run multiple scenarios to determine the "optimal" workflow. Future similar procedures can use the updated flow to increase efficiency and reduce fatigue. For example, improved logistics and surgical efficiency may be demonstrated by enabling virtual objects on an augmented reality display. Such virtual objects may include graphical overlays on images of device layouts, equipment locations, and patient layouts to visualize the flow and use of products.

[0129] Several exemplary diagrams of an optimized operating room are depicted in Figures 19A, 19B, and 19C. Each of Figures 11200a, b, c depicts a possible arrangement of the components of the operating room, including one or more support areas 11202a, b, c, transition areas 11204a, b, c (corresponding to the circulation area 11106 in Figure 17), and one or more supply areas 11206a, b, c. The anesthesia area is indicated by 11208a, b, c, and the operating tables 11210a, b, c are located in the central operating room (11118 in Figure 17). In these depictions, the operating tables 11210a, b, c are subdivided into a right table area, a left table area, and a foot area.

[0130] After the surgery is complete, an overlay is projected to indicate which disposal method should be used for the various instruments and materials in the operating room. In some examples, batteries may be directed to their own waste stream. Disposable parts of used medical devices or intelligent medical devices may be directed to the medical waste disposal area. Unused materials may be returned to stock or remain in the operating room. Packaging of medical devices or disposable medical supplies can be tracked to ensure proper recycling, reuse, or disposal.

[0131] Customization of virtual object data based on device ownership and task needs In some embodiments, the augmented reality display may be customized by its associated user to enable the display of virtual objects related to intelligent surgical instruments under the user's control. The interactive surgical system may monitor the status of intelligent surgical instruments under the control of each member of the surgical team in the operating room. The interactive surgical system may then control the display of one or more virtual objects on the augmented reality display associated with the surgical team member, which are the result of both the device under the control of the surgical team member and the task or situation in which the team member is acting.

[0132] An interactive surgical system can track which surgical team member is using which augmented reality display device. This may include conventional monitors, but may also include augmented reality glasses, wearables, secondary displays, instrument displays, and capital instrument control displays. Then, using its understanding of the instruments being used by the surgical team members, the interactive surgical system can adjust the display on the display associated with the surgical team member with virtual objects useful for the task at hand. In one embodiment, such virtual objects may appear on the augmented reality display of the surgical team member controlling the intelligent surgical device. Alternatively, such virtual objects may appear on the augmented reality displays of some or all of the members of the surgical team. This may be done for all users in the OR and all instruments and displays they are using simultaneously. In some embodiments, the virtual objects may appear as colored highlights surrounding images of surgical instruments. The highlights may appear as contours or colored overlays displayed on images of surgical instruments. Other virtual objects may include auxiliary window displays containing text messages or secondary images of tissue visible in the surgical field. In some other embodiments, the auxiliary window display may include images derived from models computed by an artificial intelligence module in a cloud system. These models may include images of tissue hidden from the view within the surgical field, or alternative views of intelligent medical instruments used by the surgeon.

[0133] In some embodiments, the augmented reality display may enable the display of virtual objects related to the state of an intelligent surgical device customized by the associated user and under the control of the associated user or another surgical team member. The state may include whether the device is powered, whether it is performing one of several types of functions, the device's power level, the state of auxiliary components such as staples, error conditions, and similar functional states.

[0134] In some embodiments, an augmented reality display may be customized by its associated user and enable the display of virtual objects related to events such as the application of another surgical device to a patient. The virtual objects may display counters or timers, such as linear or circular counters. Timers may be used to time events.

[0135] In some embodiments, an augmented reality display may enable the display of virtual objects that are customized by its associated user and relate to the state of the paired device. In one example, the virtual object may be a highlight overlaid on a stapler having a color or intensity related to the stapler's energy level. In some examples, the highlight may be applied to the augmented reality display only when the stapler is controlled by the surgeon associated with the display device.

[0136] In some embodiments, an augmented reality display may be customized by its associated user and enable the display of virtual objects related to the state of the active device. Intelligent surgical devices located in the operating room may actively communicate with (or be actively paired with) interactive surgical systems, communication hubs, cloud systems, or other equipment. Surgical devices may or may not be actively used by the surgeon, but may simply be placed on a Mayo stand ready for use. A device actively used by the surgeon may be one currently being used or held by the surgeon. A device used by the surgeon may be one in the surgical field that is currently performing some action on the surgeon.

[0137] In some embodiments, virtual objects related to the state of the active device may disappear from the display after a certain period of inactivity. In some embodiments, virtual objects related to the state of the active device may display device-related information only when the device is actively in use or in use.

[0138] In some embodiments, an augmented reality display may be customized by its associated user based on user input or requests to activate the display. Depending on the technical capabilities of the augmented reality display, input may be received from a keyboard, mouse, verbal commands, gestures, haptic input on a touch-sensitive screen, a stylus, or any other means of information input.

[0139] It can be understood that multiple augmented reality display devices may be used in the operating room. One or more primary widescreen displays may be available for use by all members of the surgical team. Alternatively, other display types may be associated with each member of the surgical team by the interactive surgical system. Such devices may include one or more laptop devices, tablet devices, or wearable devices such as augmented reality headsets. Tablet display devices may differ from larger display devices within a standard OR. If the interactive surgical system determines, or indicates, that a tablet screen is being used, virtual objects displayed on the augmented reality display device may be adjusted to fit the smaller screen in terms of display location, aspect ratio, color, or other visual design aspects. The user can determine which virtual objects should be present and which should be excluded.

[0140] A surgical team member may interact with a portion of the associated augmented reality display to determine where a particular virtual object will appear. In other embodiments, a surgical team member may scan, photograph, or input classifiers that will specify the display of an overlay, configuration, or location. In other embodiments, a surgical team member may interact with a portion of the augmented reality display via a separate device, such as a wearable device, to identify a given configuration or input and customize the layout of the augmented reality display associated with the user. In other embodiments, the user's audio or visual sources may be linked with instruments in their control. In some other embodiments, a virtual object representing linked or interactive displays of multiple instruments may be displayed together on multiple augmented reality displays or on a primary or first operating room display, with a more detailed overview than each of the individual, independent displays.

[0141] Individual members of a surgical team may be able to customize the display of virtual objects on associated augmented reality displays, but the information displayed for a particular surgical team member may change as intelligent surgical instruments begin or end their control. In some embodiments, the intelligent surgical system may track instrument exchanges and changes in ownership among members of the surgical team and adjust the displayed augmented reality data based on the changes in ownership. Furthermore, the interactive surgical system can determine the original position and out-of-body configuration of the instrument in use, as well as the associated user controlling the instrument. To verify the correct correlation between the two facets, the interactive surgical system can correlate actions occurring out-of-body with movements occurring in-body.

[0142] Various additional aspects of the subject matter described herein are illustrated in the following numbered examples.

[0143] Example 1: A method for distributing data among members of a surgical team, comprising: receiving imaging data from multiple imaging devices by a modular control tower; receiving device-dependent data from each of multiple intelligent surgical instruments by a modular control tower; associating display devices with members of the surgical team by a modular control tower; defining the functional roles of the members of the surgical team by a modular control tower; and displaying an augmented reality display by a display device by a modular control tower, wherein the augmented reality display on the display device includes imaging data, device-dependent data, the functional roles of the members of the surgical team, and virtual objects based on surgical activities by the members of the surgical team.

[0144] Example 2: The method according to Example 1, wherein the modular control tower receives device-dependent data, and the modular control tower receives data defining members of a surgical team who control one or more of several intelligent surgical instruments.

[0145] Example 3: The method according to Example 2, further comprising displaying virtual objects associated with the control function of one or more of the multiple intelligent surgical instruments by the surgical team member on an augmented reality display associated with the surgical team member controlling one or more of the multiple intelligent surgical instruments via a modular control tower.

[0146] Example 4: The method according to Example 3, further comprising: a modular control tower determining one or more in-situ and out-of-body configurations of a plurality of intelligent surgical instruments controlled by members of a surgical team; and the modular control tower displaying virtual objects associated with one or more in-situ configurations of the plurality of intelligent surgical instruments on a display device associated with a member of the surgical team controlling one or more of the plurality of intelligent surgical instruments.

[0147] Example 5: The method according to Example 4, further comprising using a modular control tower to correlate the operation of one or more in-situ modes of a plurality of intelligent surgical instruments with the operation of one or more extracorporeal modes of a plurality of intelligent surgical instruments.

[0148] Example 6: The method according to Example 3, further comprising the modular control tower changing a virtual object displayed on an augmented reality display associated with the member of the surgical team who controls one or more of the intelligent surgical instruments when the member of the surgical team relinquishes control of one or more of the intelligent surgical instruments.

[0149] Example 7: The method according to Example 6, further comprising changing an augmented reality display on a display device associated with a member of a surgical team receiving control of one or more of several intelligent surgical instruments by a modular control tower.

[0150] Example 8: The method according to Example 1, further comprising causing a first member of the surgical team to display an augmented reality display associated with a second member of the surgical team on a display device associated with a second member of the surgical team.

[0151] Example 9: The method according to Example 1, further comprising causing a first member of the surgical team to display an augmented reality display associated with a second member of the surgical team on a display device associated with the first member of the surgical team.

[0152] Example 10: The method according to Example 1, further comprising adjusting a virtual object on an augmented reality display associated with a member of the surgical team by a member of the surgical team.

[0153] Example 11: The method according to Example 1, further comprising adjusting the appearance of a virtual object displayed on an augmented reality display associated with a member of the surgical team.

[0154] Example 12: The method according to Example 11, wherein adjusting the appearance of a virtual object displayed on an augmented reality display includes adjusting the location of one or more of the virtual objects on the augmented reality display.

[0155] Example 13: An automated surgical system comprising a modular control tower, a plurality of imaging devices communicating with the modular control tower, a plurality of intelligent surgical instruments, and a plurality of display devices communicating with the modular control tower. Each of the plurality of display devices is associated with one or more members of a surgical team by the modular control tower, and each of the one or more members of the surgical team is defined by a functional role. The modular control tower comprises a controller that communicates with one or more memory components configured to store instructions, and when executed by the controller, the instructions cause the controller to receive imaging data from the plurality of imaging devices, receive device-dependent data from each of the plurality of intelligent surgical instruments, and display an augmented reality display on each of the plurality of display devices. The augmented reality display on a designated display device may include imaging data, device-dependent data, the functional role of a designated member of the surgical team associated with the designated display device, and virtual objects based on surgical activities by the designated member of the surgical team.

[0156] Example 14: The system according to Example 13, wherein each of the augmented reality displays of the multiple display devices is the same.

[0157] Example 15: The system according to Example 13, wherein the augmented reality display of a specified display device is customizable by a specified member of the surgical team associated with the specified display device.

[0158] Example 16: The system according to Example 13, wherein the virtual objects of the augmented reality display depend on the type of one or more display devices.

[0159] Example 17: The system according to Example 13, wherein a virtual object of an augmented reality display on a designated display device depends on one or more intelligent surgical instruments controlled by a designated member of a surgical team.

[0160] Example 18: The system according to Example 17, wherein a virtual object on an augmented reality display changes when a designated member of the surgical team relinquishes control of one or more intelligent surgical instruments.

[0161] Example 19: The system according to Example 17, wherein virtual objects on an augmented reality display on a designated display device include predictions of a second surgical activity by a designated member of the surgical team.

[0162] Example 20: The system according to Example 17, wherein a virtual object on an augmented reality display depends on the distance of one or more intelligent surgical instruments from a patient's important anatomical structure.

[0163] Example 21: The system according to Example 17, wherein a virtual object on an augmented reality display indicates that one or more intelligent surgical instruments, controlled by a designated member of a surgical team, are being used improperly or are unsuitable for a surgical activity.

[0164] Example 22: The system according to Example 17, wherein a virtual object on an augmented reality display depicts a view of one or more intelligent surgical instruments that are not visible to a designated member of the surgical team.

[0165] While several forms have been shown and described, it is not the applicant's intention to limit or restrict the attached claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be implemented and will be conceived by those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element related to the described form can be alternatively described as a means for providing the function performed by that element. Also, while materials are disclosed with respect to specific components, other materials may be used. Therefore, it should be understood that the above description and the attached claims are intended to cover all such modifications, combinations, and variations as being included within the scope of the disclosed forms. The attached claims are intended to cover all such modifications, variations, alterations, substitutions, alterations, and equivalents.

[0166] The detailed descriptions above have described various forms of apparatus and / or processes using block diagrams, flowcharts and / or embodiments. To the extent that such block diagrams, flowcharts and / or embodiments include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flowcharts and / or embodiments can be implemented individually and / or collectively by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some or all of the forms disclosed herein can be equivalently implemented on integrated circuits as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code falls within the scope of the skills of those skilled in the art in light of this disclosure. Furthermore, as will be understood by those skilled in the art, the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and the specific forms of the subject matter described herein are applicable regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0167] Instructions used to program logic to implement various disclosed embodiments may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, instructions may be distributed over a network or by other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but are not limited to floppy diskettes, optical disks, compact disks, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage used for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, non-temporary computer-readable media may include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0168] When used in any aspect of this specification, the term “control circuit” can mean, for example, hardwired circuits, programmable circuits (e.g., computer processors, processing units, processors, microcontrollers, microcontroller units, controllers, digital signal processors (DSPs), programmable logic devices (PLDs), programmable logic arrays (PLAs), or field-programmable gate arrays (FPGAs) including one or more individual instruction processing cores), state-machine circuits, firmware that stores instructions executed by programmable circuits, and any combination thereof. Control circuits can be embodied collectively or individually as circuits that form part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, or a smartphone. Accordingly, as used herein, “control circuit” includes, but is not limited to, an electrical circuit having at least one separate electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application-specific integrated circuit, an electrical circuit forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program that performs at least partially the processes and / or devices described herein, or a microprocessor configured by a computer program that performs at least partially the processes and / or devices described herein), an electrical circuit forming a memory device (e.g., in the form of random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electric installation). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog form, digital form, or some combination thereof.

[0169] When used in any aspect of this specification, the term “logic” may mean an application, software, firmware, and / or circuit configured to perform any of the operations described above. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-temporary computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and / or hardcoded (e.g., non-volatile) data in a memory device.

[0170] When used in any aspect of this specification, the terms “component,” “system,” “module,” etc., may refer to a control circuit, a computer-related entity, hardware, a combination of hardware and software, software, or running software.

[0171] Where used in any aspect of this specification, “algorithm” means a self-consistent sequence of steps leading to a desired result, and “step” means the manipulation of physical quantities and / or logical states that can take the form of electrical or magnetic signals, which are not necessarily required but can be stored, transferred, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities, or simply are convenient labels applied to these quantities and / or states.

[0172] A packet-switched network is one example of a network. Communication devices can communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol is the Ethernet communication protocol, which can enable communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of the Ethernet standard. Alternatively or additionally, communication devices can communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices can communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol conforms to or may be compatible with standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol conforms to or may be compatible with the ATM standard and / or later versions of this standard, published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0". Naturally, different and / or later developed connection-oriented network communication protocols are equally construed herein.

[0173] Unless otherwise explicitly stated, as is evident from the foregoing disclosures, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the foregoing disclosures should be understood to refer to the operation and processing of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage, transmission, or display device.

[0174] One or more components may be referred to herein as “configured to,” “configurable to,” “operable / operative to,” “adapted / adaptable,” “able to,” “conformable / conformed to,” and so on. Those skilled in the art will understand that “configured to” generally encompasses active components and / or inactive components and / or standby components, unless the context should interpret it otherwise.

[0175] The terms “proximal” and “distal” are used herein in reference to the clinician operating the handle portion of a surgical instrument. “Proximal” refers to the part closest to the clinician, and “distal” refers to the part further away from the clinician. For convenience and clarity, spatial terms such as “vertical,” “horizontal,” “up,” and “down” may be used herein in reference to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0176] Those skilled in the art will generally understand that the terms used herein, and especially in the appended claims (e.g., the text of the appended claims), are generally intended to be "open" terms (for example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "includes" should be interpreted as "includes but is not limited to"). Furthermore, those skilled in the art will understand that if a particular number is intended in an introduced claim recitation, such intent is clearly stated in the claim, and if such statement is not present, such intent does not exist. For example, to aid understanding, subsequent appended claims may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be interpreted as suggesting that any particular claim containing such introduced claim description is limited to claims containing only one such description, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" (for example, "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using a definite article.

[0177] In addition, even if a specific number is explicitly stated in the introduced claim, it will be recognized by those skilled in the art that such a statement should typically be interpreted as meaning at least the number stated (for example, if there is a statement that is simply “two descriptions” without any other modifiers, it generally means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to “at least one of A, B, and C, etc.” is used, such a notation is generally intended to be understood in a way that those skilled in the art will understand (for example, “a system having at least one of A, B, and C” is not limited to systems having only A, only B, only C, both A and B, both A and C, both B and C and / or all of A, B and C, etc.). When expressions similar to "at least one of A, B, or C" are used, such expressions are generally intended to be understood in a way that a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Furthermore, a person skilled in the art will understand that, typically, any disjunctive word and / or phrase representing two or more selective terms should be understood, whether in the specification, claims, or drawings, as intended to include the possibility of including one of those terms, any of those terms, or both of those terms, unless the context requires a different interpretation. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0178] With respect to the attached claims, those skilled in the art will understand that the operations cited herein may generally be performed in any order. Furthermore, while various operations are shown in sequence(s), it should be understood that the operations may be performed in any order other than those shown, or simultaneously. Examples of such alternative orderings may include repetition, alternation, interruption, reordering, augmentation, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context should imply otherwise. Moreover, terms such as “responsive to,” “related to,” or other past tense adjectives are generally not intended to exclude such variations, unless the context should imply otherwise.

[0179] It is worth noting that any reference to “one aspect,” “aspect,” “example,” or “example” means that the specific feature, structure, or characteristic described in relation to that aspect is included in at least one aspect. Therefore, the phrases “in one aspect,” “in aspect,” “example,” and “example” found in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, specific features, structures, or characteristics can be combined in any preferred manner in one or more aspects.

[0180] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material does not conflict with this Specified. Disclosures expressly stated herein, both in themselves and to the extent required, shall supersede any conflicting statements incorporated herein by reference. Any material, or any part thereof, that is referred to as being incorporated herein by reference but conflicts with current definitions, views, or other disclosures contained herein shall be incorporated only to the extent that there is no conflict between the incorporated material and the current disclosures.

[0181] In summary, the numerous benefits that can be obtained as a result of using the concepts described herein have been described. The above descriptions of one or more forms are presented for illustrative and explanatory purposes only. They are not intended to be comprehensive or to be limited to the exact forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms have been selected and described to illustrate the principle and practical applications, thereby enabling a person skilled in the art to utilize the various forms, along with various modifications, for specific conceivable uses. The claims presented herein are intended to define the overall scope.

[0182] [Implementation Method] (1) A method for distributing data among members of a surgical team, The modular control tower allows for the reception of imaging data from multiple imaging devices, The modular control tower receives device-dependent data from each of the multiple intelligent surgical instruments, The modular control tower allows for the association of display devices with members of the surgical team, The modular control tower defines the functional roles of the members of the surgical team, The modular control tower includes displaying an augmented reality display using the display device, A method wherein the augmented reality display on the display device includes the imaging data, the device-dependent data, the functional roles of the members of the surgical team, and virtual objects based on surgical activities performed by the members of the surgical team. (2) The method according to Embodiment 1, wherein the modular control tower receives device-dependent data, and the modular control tower receives data defining members of the surgical team who control one or more of the plurality of intelligent surgical instruments. (3) The method of Embodiment 2, further comprising displaying virtual objects of one or more of the intelligent surgical instruments, associated with the control function by the member of the surgical team, on an augmented reality display associated with the member of the surgical team controlling one or more of the plurality of intelligent surgical instruments via the modular control tower. (4) The modular control tower determines one or more in-situ and out-of-body configurations of the plurality of intelligent surgical instruments controlled by the members of the surgical team, The method according to Embodiment 3, further comprising the modular control tower displaying virtual objects associated with one or more of the in-situ configurations of the plurality of intelligent surgical instruments on a display device associated with the member of the surgical team controlling one or more of the plurality of intelligent surgical instruments. (5) The method according to Embodiment 4, further comprising using the modular control tower to correlate the operation of one or more of the in-situ modes of the plurality of intelligent surgical instruments with the operation of one or more of the plurality of intelligent surgical instruments in extracorporeal modes.

[0183] (6) The method according to Embodiment 3, further comprising, when the member of the surgical team relinquishes control of one or more of the plurality of intelligent surgical instruments, the modular control tower modifies the virtual object displayed on the augmented reality display associated with the member of the surgical team who controls one or more of the plurality of intelligent surgical instruments. (7) The method according to embodiment 6, further comprising changing an augmented reality display on a display device associated with a member of the surgical team who receives control of one or more of the plurality of intelligent surgical instruments, by the modular control tower. (8) The method according to Embodiment 1, further comprising causing a first member of the surgical team to cause a display device associated with a second member of the surgical team to display an augmented reality display associated with the first member of the surgical team. (9) The method according to Embodiment 1, further comprising causing a first member of the surgical team to display an augmented reality display associated with a second member of the surgical team on a display device associated with the first member of the surgical team. (10) The method according to Embodiment 1, further comprising adjusting the virtual object of the augmented reality display associated with the member of the surgical team.

[0184] (11) The method according to Embodiment 1, further comprising adjusting the appearance of the virtual object displayed on the augmented reality display associated with the member of the surgical team. (12) The method according to Embodiment 11, wherein adjusting the appearance of the virtual objects displayed on the augmented reality display includes adjusting the location of one or more of the virtual objects on the augmented reality display. (13) An automated surgical system, Modular control tower and Multiple imaging devices that communicate data with the aforementioned modular control tower, Multiple intelligent surgical instruments, The modular control tower comprises a plurality of display devices that communicate with each other, Each of the plurality of display devices is associated with one or more members of the surgical team by the modular control tower, and each of the one or more members of the surgical team is defined by a functional role. The modular control tower includes a controller that communicates data with one or more memory components configured to store instructions, and when an instruction is executed by the controller, the controller Receiving imaging data from the aforementioned multiple imaging devices, Receiving device-dependent data from each of the aforementioned multiple intelligent surgical instruments, An automated surgical system that displays an augmented reality display on each of the plurality of display devices, wherein the augmented reality display on a designated display device includes the imaging data, the device-dependent data, the functional roles of designated members of the surgical team associated with the designated display device, and virtual objects based on surgical activities performed by the designated members of the surgical team. (14) The system according to embodiment 13, wherein each of the plurality of display devices is the same augmented reality display. (15) The system according to Embodiment 13, wherein the augmented reality display of the designated display device is customizable by the designated member of the surgical team associated with the designated display device.

[0185] (16) The system according to embodiment 13, wherein the virtual objects of the augmented reality display depend on the type of one or more display devices. (17) The system according to Embodiment 13, wherein the virtual objects of the augmented reality display on the designated display device depend on one or more intelligent surgical instruments controlled by the designated member of the surgical team. (18) The system according to Embodiment 17, wherein the virtual object on the augmented reality display changes when the designated member of the surgical team relinquishes control of the one or more intelligent surgical instruments. (19) The system according to Embodiment 17, wherein the virtual objects of the augmented reality display on the designated display device include predictions of a second surgical activity by the designated member of the surgical team. (20) The system according to Embodiment 17, wherein the virtual object of the augmented reality display depends on the distance of the one or more intelligent surgical instruments from the patient's important anatomical structures.

[0186] (21) The system according to Embodiment 17, wherein the virtual object on the augmented reality display indicates that one or more intelligent surgical instruments controlled by the designated member of the surgical team are being used improperly or are unsuitable for the surgical activity. (22) The system according to Embodiment 17, wherein the virtual object on the augmented reality display depicts a view of one or more intelligent surgical instruments that are not visible to the designated member of the surgical team.

Claims

1. A surgical system, Modular control tower and Multiple imaging devices that communicate data with the aforementioned modular control tower, Multiple intelligent surgical instruments, The modular control tower comprises a plurality of display devices that communicate with each other, Each of the plurality of display devices is associated with one or more members of the surgical team by the modular control tower, and each of the one or more members of the surgical team is defined by a functional role. The modular control tower includes a controller that communicates data with one or more memory components configured to store instructions, and when an instruction is executed by the controller, the controller Receiving imaging data from the aforementioned multiple imaging devices, Receiving device-dependent data from each of the aforementioned multiple intelligent surgical instruments, A surgical system that displays augmented reality visual information on each of the plurality of display devices, wherein the augmented reality visual information on a designated display device includes the imaging data, the device-dependent data, the functional roles of designated members of the surgical team associated with the designated display device, and virtual objects based on surgical activities performed by the designated members of the surgical team.

2. The system according to claim 1, wherein the augmented reality visual information of each of the plurality of display devices is the same.

3. The system according to claim 1, wherein the augmented reality visual information of the designated display device is customizable by the designated member of the surgical team associated with the designated display device.

4. The system according to claim 1, wherein the virtual objects of the augmented reality visual information depend on the type of the plurality of display devices.

5. The system according to claim 1, wherein the virtual objects of the augmented reality visual information on the designated display device depend on one or more intelligent surgical instruments controlled by the designated member of the surgical team.

6. The system according to claim 5, wherein the virtual object of the augmented reality visual information changes when one or more intelligent surgical instruments are out of the control of the designated member of the surgical team.

7. The system according to claim 5, wherein the virtual objects of the augmented reality visual information on the designated display device include predictions of a second surgical activity by the designated member of the surgical team.

8. The system according to claim 5, wherein the virtual objects in the augmented reality visual information depend on the distance of one or more intelligent surgical instruments from important anatomical structures of the patient.

9. The system according to claim 5, wherein the virtual object of the augmented reality visual information indicates that one or more intelligent surgical instruments controlled by the designated member of the surgical team are being used improperly or are unsuitable for the surgical activity.

10. The system according to claim 5, wherein the virtual objects in the augmented reality visual information depict a view of one or more intelligent surgical instruments that are not visible to the designated members of the surgical team.

11. A method for distributing data among members of a surgical team, The modular control tower allows for the reception of imaging data from multiple imaging devices, The modular control tower receives device-dependent data from each of the multiple intelligent surgical instruments, The modular control tower allows for the association of display devices with members of the surgical team, The modular control tower defines the functional roles of the members of the surgical team, The modular control tower includes displaying augmented reality visual information on the display device, A method wherein the augmented reality visual information on the display device includes the imaging data, the device-dependent data, the functional roles of the members of the surgical team, and virtual objects based on surgical activities performed by the members of the surgical team.

12. The method according to claim 11, wherein the modular control tower receives device-dependent data, and the modular control tower receives data defining members of the surgical team who control one or more of the plurality of intelligent surgical instruments.

13. The method according to claim 12, further comprising displaying virtual objects associated with the control function of one or more of the plurality of intelligent surgical instruments by the member of the surgical team on augmented reality visual information associated with the member of the surgical team controlling one or more of the plurality of intelligent surgical instruments, using the modular control tower.

14. The method according to claim 13, further comprising, when one or more of the plurality of intelligent surgical instruments is out of the control of the member of the surgical team, the modular control tower modifies the virtual object displayed on the augmented reality visual information associated with the member of the surgical team controlling one or more of the plurality of intelligent surgical instruments.

15. The method according to claim 14, further comprising modifying augmented reality visual information on a display device associated with a member of the surgical team receiving control of one or more of the plurality of intelligent surgical instruments by the modular control tower.

16. The method according to claim 11, further comprising causing a first member of the surgical team to display augmented reality visual information associated with the first member of the surgical team on a display device associated with a second member of the surgical team.

17. The method according to claim 11, further comprising causing a first member of the surgical team to display augmented reality visual information associated with a second member of the surgical team on a display device associated with the first member of the surgical team.

18. The method according to claim 11, further comprising adjusting the virtual object of the augmented reality visual information associated with the member of the surgical team.

19. The method according to claim 11, further comprising adjusting the appearance of the virtual object displayed on the augmented reality visual information associated with the member of the surgical team.

20. The method according to claim 19, wherein adjusting the appearance of the virtual objects displayed on the augmented reality visual information includes adjusting the location of one or more of the virtual objects on the augmented reality visual information.

Images