Mixed elements and actions occurring on and off screen are directly visualized in rendered elements.

Abstract

Apparatus, systems, and methods for mixed reality visualization are disclosed herein. In one aspect, a method for mixed reality visualization includes capturing an image of an object within a surgical field by a first camera of a first visualization system, where a first portion of the object is outside of a field of view of the first camera, tracking a position of a second portion of the object by a tracking system, determining, by a surgical hub, an attribute of the object based on the tracked position of the second portion of the object, where the attribute of the object is associated with the first portion of the object outside of the field of view of the camera, and displaying, by an augmented reality display device, the captured image of the object within the surgical field and a graphic based on the attribute of the object.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 174,674, filed on April 14, 2021, entitled "HEADS UP DISPLAY", and U.S. Provisional Patent Application No. 63 / 284,326, filed on November 30, 2021, entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS", under 35 U.S.C. § 119(e). The entire disclosure of each of these applications is hereby incorporated by reference in its entirety.

Background Art

[0002] The present disclosure relates to devices, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure, it would be desirable to provide an augmented reality interactive experience of the real - world environment in which objects existing in the real world are enhanced by overlaying computer - generated perceptual information across multiple sensory modalities, including, at times, vision, hearing, touch, proprioception, and smell. In the context of the present disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer - generated visual, auditory, tactile, proprioceptive, olfactory, or other sensory information onto the real - world images of the surgical field and the instruments or other objects appearing in the surgical field. The images may be streamed in real - time or may be still images.

[0003] Real-world surgical instruments include a variety of surgical devices, including energy, staplers, or combinations of energy and staplers. Energy-based medical devices include, but are not limited to, radio frequency (RF) based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including obesity, thoracic, colorectal, obstetric and gynecological, urological, and general surgery. [Overview of the Initiative] [Means for solving the problem]

[0004] In various embodiments, the Disclosure provides a method for augmented reality visualization of a surgical system. In some embodiments, the method includes capturing an image of an object in the surgical field with a first camera of a first visualization system, wherein a first portion of the object is outside the field of view of the first camera; tracking the position of a second portion of the object with a tracking system; determining the attributes of the object based on the tracked position of the second portion of the object with a surgical hub, wherein the attributes of the object relate to the first portion of the object outside the field of view of the camera; and displaying the captured image of the object in the surgical field and graphics based on the attributes of the object with an augmented reality display device. In one embodiment, the object includes a surgical instrument, patient tissue, or a user, or a combination thereof.

[0005] In various embodiments, the Disclosure provides a surgical system for augmented reality visualization. In some embodiments, the system comprises: a first visualization system comprising a first camera configured to capture an image of an object in the surgical field, wherein a first portion of the object is outside the camera's field of view; a first tracking system configured to track the position of a second portion of the object; a surgical hub configured to determine the attributes of an object based on the tracked position of the second portion of the object, wherein the attributes of the object are related to a first portion of the object outside the camera's field of view; and an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object. In one embodiment, the object includes a surgical instrument, patient tissue, a user, or a combination thereof. [Brief explanation of the drawing]

[0006] The various embodiments described herein with respect to both configuration and operation methods, along with their further purposes and advantages, can be best understood by referring to the following description in conjunction with the accompanying drawings. [Figure 1] This is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 2] This is a diagram of a surgical system used to perform surgical procedures in an operating room, according to one aspect of the present disclosure. [Figure 3] One aspect of the present disclosure is a visualization system, a robotic system, and a surgical hub paired with an intelligent instrument. [Figure 4] This figure shows a surgical data network, according to one aspect of the present disclosure, which includes a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5]This figure shows a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 6] This figure shows a surgical hub, including multiple modules connected to a modular control tower, according to one aspect of the present disclosure. [Figure 7] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 8] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 9] This figure shows an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. [Figure 10] This figure shows a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. [Figure 11] This figure shows a timeline of a situational awareness surgical procedure according to one aspect of the present disclosure. [Figure 12] This figure shows a surgical system, according to one aspect of the present disclosure, which includes a tracking system configured to track objects within an operating room. [Figure 13] This is a schematic side view of an exemplary implementation of the tracking system in the operating room shown in Figure 12, according to one aspect of the present disclosure. [Figure 14] This is a schematic plan view of an exemplary operating room map generated by an operating room mapping module according to one aspect of the present disclosure. [Figure 15] An exemplary table of tracked object interactions determined by a surgical hub based on data generated by a tracking system, according to one aspect of the present disclosure. [Figure 16A] This figure shows an exemplary intraoperative display according to one aspect of the present disclosure, which includes images of surgical instruments within the surgical field and graphics representing a portion of the surgical instruments outside the field of view. [Figure 16B] This figure shows an exemplary intraoperative display according to one aspect of the present disclosure, which includes images of surgical instruments within the surgical field and graphics representing a portion of the surgical instruments outside the field of view. [Figure 17A] This figure shows an exemplary intraoperative display, according to one aspect of the present disclosure, which includes an image of gastric tissue as a surgeon uses an endocutter to make a cutting line in the gastric tissue. [Figure 17B] This figure shows an exemplary intraoperative display, according to one aspect of the present disclosure, which includes an image of gastric tissue as a surgeon uses an endocutter to make a cutting line in the gastric tissue. [Figure 18] This figure shows a method for augmented reality visualization of a surgical system according to one aspect of the present disclosure.

[0007] Throughout the drawings, corresponding reference numerals indicate corresponding parts. The examples described herein illustrate various disclosed embodiments in one form, and such examples should not be construed as limiting the scope. [Modes for carrying out the invention]

[0008] The applicant of this application owns the following concurrently filed U.S. patent applications, the entirety of which is incorporated herein by reference: • U.S. Patent Application titled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Agent Reference Number END9352USNP1 / 210120-1M • U.S. Patent Application entitled "Utilization of surgical data values ​​and situational awareness to control the overlay in surgical field view"; Agent reference number END9352USNP2 / 210120-2 U.S. Patent Application entitled "SELECTIVE AND ADJUSTABLE MIXED REALITY OVERLAY IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP3 / 210120-3 • U.S. Patent Application entitled "RISK BASED PRIORITIZATION OF DISPLAY ASPECTS IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP4 / 210120-4 • U.S. Patent Application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Agent Reference Number END9352USNP5 / 210120-5 U.S. Patent Application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Agent Reference Number END9352USNP6 / 210120-6 • U.S. Patent Application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; Agent Reference Number END9352USNP7 / 210120-7 U.S. Patent Application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; Agent Reference Number END9352USNP8 / 210120-8 U.S. Patent Application entitled "Cooperative Overlays of Interacting Instruments Which Resurface in Both Overlays Being Effected"; Agent Reference Number END9352USNP9 / 210120-9 · U.S. patent application titled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Attorney Docket No. END9352USNP10 / 210120-10, · U.S. patent application titled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Attorney Docket No. END9352USNP12 / 210120-12, · U.S. patent application titled "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION"; Attorney Docket No. END9352USNP13 / 210120-13, · U.S. patent application titled "COOPERATION AMONG MULTIPLE DISPLAY SYSTEMS TO PROVIDE A HEALTHCARE USER CUSTOMIZED INFORMATION"; Attorney Docket No. END9352USNP14 / 210120-14, · U.S. patent application titled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP15 / 210120-15, · U.S. patent application titled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS"; Attorney Docket No. END9352USNP16 / 210120-16, and, · U.S. Patent Application titled "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS"; Attorney Docket No. END9352USNP17 / 210120-17.

[0009] The applicant of this application owns the following U.S. patent applications, the entire disclosure of each of which is incorporated herein by reference. · U.S. Patent Application No. 16 / 209,423 (now U.S. Patent Application Publication No. 2019 / 0200981-A1) titled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", · U.S. Patent Application No. 16 / 209,453 (now U.S. Patent Application Publication No. 2019 / 0201046-A1) titled "METHOD FOR CONTROLLING SMART ENERGY DEVICES".

[0010] Before describing various aspects of the surgical device and generator in detail, it should be noted that the exemplary embodiments are not limited to the details of the structure and arrangement of the components illustrated in the accompanying drawings and description in terms of application or use. The exemplary embodiments may be implemented in or incorporated into other aspects, variations, and modifications, and may be implemented or executed in various ways. Further, unless otherwise specified, the terms and expressions used herein are selected for the purpose of explaining the exemplary embodiments for the convenience of the reader and are not intended to limit them. Further, it should be understood that one or more of the aspects, expressions of aspects, and / or embodiments described below can be combined with any one or more of the other aspects, expressions of aspects, and / or embodiments described below.

[0011] Various embodiments apply to on-screen displays for surgical systems for various energy and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio frequency (RF) based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices include surgical staplers combined with electrosurgical devices and / or ultrasonic devices. Embodiments of ultrasonic surgical devices may be configured, for example, to transversely incise and / or coagulate tissue during surgical procedures. Embodiments of electrosurgical devices may be configured, for example, to transversely incise, coagulate, seal, weld and / or dry tissue during surgical procedures. Embodiments of surgical stapler devices may be configured to transversely incise and staple tissue during surgical procedures, and in some embodiments, surgical stapler devices may be configured to deliver RF energy to tissue during surgical procedures. Electrosurgical devices are configured to deliver therapeutic and / or non-therapeutic RF energy to tissue. Elements of surgical staplers, electrosurgical devices, and ultrasound devices can be used in combination within a single surgical instrument.

[0012] In various embodiments, the Disclosure provides the OR team with on-screen displays of real-time information during surgical procedures. According to various embodiments of the Disclosure, many novel and unique on-screen displays are provided for displaying various visual information feedback to the OR team on screen. According to the Disclosure, the visual information may include one or more of various visual media, with or without sound. Generally, the visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. The visual information may be communicated on any number of display options, such as, for example, a primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, etc.

[0013] In various embodiments, this disclosure provides a list of many potential options for communicating visual information to an OR team in real time without overwhelming the OR team with too much visual information. For example, in various embodiments, this disclosure provides on-screen displays of visual information that enable a surgeon, or other members of the OR team, to selectively activate on-screen displays, such as icons surrounding screen options, to manage the rich visual information. One or a combination of factors may be used to determine the active display, and these may include, among other things, the energy-based (e.g., electrosurgery, ultrasound) or machine-based (e.g., stapler) surgical device in use, the estimated risk associated with a given display, the surgeon's level of experience, and the surgeon's choice. In other embodiments, the visual information may include rich data overlaid or superimposed on the surgical field to manage the visual information. In various embodiments described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. Visual information data thus communicated can provide additional useful visual information to the OR team in a more concise and understandable way, in contrast to static icons.

[0014] In various embodiments, the Disclosure provides techniques for selectively activating on-screen displays, such as icons surrounding a screen, to manage visual information during a surgical procedure. In other embodiments, the Disclosure provides techniques for determining an active display using one or a combination of factors. In various embodiments, the techniques provided by the Disclosure may include, among other things, selecting an energy-based or machine-based surgical device to be used as the active display, estimating the risks associated with a given display, and utilizing the experience level of the surgeon or OR team making the selection.

[0015] In other embodiments, the techniques described herein may include overlaying or superimposing rich data onto the surgical field for the purpose of managing visual information. Several display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Graphical overlays may take the form of transparent graphics, translucent graphics, or opaque graphics, or combinations of transparent, translucent, and opaque elements or effects. Furthermore, overlays may be positioned on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values. Graphical overlays are rendered on an active display monitor to quickly and efficiently communicate critical information to the OR team.

[0016] In other embodiments, the technology provided by the Disclosure may include superimposing images that require video analysis and tracking in order to appropriately overlay visual information data. In other embodiments, the technology provided by the Disclosure may include communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easily understandable manner. In other embodiments, the visual overlay may be used in combination with auditory and / or somatosensory overlays, e.g., thermal, chemical, and mechanical devices, and combinations thereof.

[0017] The following description generally pertains to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information onto the real-world images of the surgical field, instruments, and / or other objects appearing in the surgical field. The images may be streamed in real time or they may be still images. Augmented reality is a technique for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid on a real environment. The real environment may include the surgical field. Virtual objects overlaid on a real environment may be represented at fixed or set positions relative to one or more aspects of the real environment. In non-limiting examples, if a real-world object moves out of the field of view of the real environment, the virtual object fixed to the real-world object also moves out of the field of view of augmented reality.

[0018] Some of the display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value.

[0019] As described herein, AR is an extended version of the real physical world achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear real, making the user feel immersed in them. This environment is perceived through a device known as a virtual reality headset or helmet. While both mixed reality (MR) and AR are considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. AR often adds digital elements to a live view by using a camera, while an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

[0020] In an AR environment, one or more computer-generated virtual objects may be displayed alongside one or more real-world (i.e., so-called "real-world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlay virtual objects. Such virtual objects can provide supplementary information about the environment or, in general, enhance the user's perception and engagement with the environment. Conversely, real-time images or videos of the surrounding environment can, in addition or alternatively, enhance the user's engagement with the virtual objects displayed on the display.

[0021] Apparatus, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during surgical procedures. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and / or cameras that provide images during incisional surgical procedures. Images may be streamed in real time or still images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing images of a real-world surgical environment by overlaying representations of virtual objects or data and / or real objects onto the real-world surgical environment. The augmented reality experience may be viewed on a display and / or AR device that allows the user to view virtual objects overlaid on the real-world surgical environment. The display may be located in the operating room or located away from the operating room. The AR device is worn on the head of a surgeon or other operating room personnel and typically includes two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through the two transparent or translucent display lenses so that aspects of the real environment are visible, while projecting light to make virtual objects visible to the user of the AR device.

[0022] Two or more displays and AR devices may be used in conjunction with a first display or AR device that controls one or more additional displays or AR devices in a system having defined roles. For example, when activating a display or AR device, the user may select a role (e.g., a surgeon, surgical assistant, nurse, etc. during a surgical procedure), and the display or AR device may display information related to that role. For example, a surgical assistant may have the display show virtual representations of instruments that the surgeon needs to use for the next step in the surgical procedure. The surgeon's focus on the current step may differ from the information displayed by the surgical assistant.

[0023] While many known on-screen displays and alerts exist, this disclosure provides many novel and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment may be provided to the operating room (OR) team, including, but not limited to, personnel within the OR, such as the surgical surgeon, surgical assistants, scrub wearers, anesthesiologists, and circulating nurses. The virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical staplers, tablets, augmented reality glasses, and other devices.

[0024] Figure 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 connected to remote storage 5. Each surgical system 2 comprises at least one surgical hub 6 that communicates with the cloud 4. For example, a surgical system 2 may comprise a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12, each configured to communicate with each other and / or with the hub 6. In some embodiments, a surgical system 2 may comprise M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12, where M, N, O, and P are integers of 1 or more. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during surgical procedures, as described herein.

[0025] Figure 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. A robotic system 10 is used as part of the surgical system 2 in the surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient-side cart 20 (surgical robot), and a surgical robot hub 22. The patient-side cart 20 allows the surgeon to operate at least one detachably connected surgical tool 17 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. Images of the surgical site during the minimally invasive procedure (e.g., still or live images streamed in real time) can be acquired by a medical imaging device 24. The patient-side cart 20 can operate the imaging device 24 to orient it. Images of the incision surgical procedure can be acquired by a medical imaging device 96. The robot hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18, or on an AR device 66 worn by the surgeon or another person in the surgical room 16.

[0026] The optical components of the imaging device 24, 96, or AR device 66 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from tissues and instruments in the surgical field.

[0027] In various embodiments, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes. In various embodiments, the imaging device 96 is configured for use in incisional (invasive) surgical procedures.

[0028] In various embodiments, the visualization system 8 comprises one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more displays. In one embodiment, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one embodiment, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range in the electromagnetic spectrum. Wavelengths are separated by filters or by instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring allows the surgical field to be repositioned after the surgical task for performing tests on the treated tissue is completed.

[0029] Figure 2 shows a primary display 19 positioned in the sterile field for the operator on the operating table 14 to see. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing opposite directions from each other. The visualization system 8, guided by the hub 6, is configured to utilize displays 7, 9, and 19 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by imaging devices 24 and 96 through the non-sterile displays 7, 9, or AR device 66, while maintaining live video of the surgical site on the primary display 19 or AR device 66. The non-sterile displays 7 and 9 can, for example, enable non-sterile operators to perform diagnostic steps related to the surgical procedure.

[0030] Figure 3 shows a hub 6 that communicates with a visualization system 8, a robotic system 10, and handheld intelligent surgical instruments 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke extraction module 26 and / or a suction / irrigation module 28. In various embodiments, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in Figure 10). Modular light sources can be adapted for use with various imaging devices. In various examples, multiple imaging devices can be positioned at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or incisional surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide the optimal view. In various embodiments, the imaging module 38 can be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during surgical procedures as described herein.

[0031] Figure 4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located in one or more operating rooms / surgery sites of a medical facility to a cloud-based system. The cloud 54 may include a remote server 63 (Figure 5) connected to a storage device 55. The modular communication hub 53 includes a network hub 57 and / or a network switch 59 that communicate with a network router 61. The modular communication hub 53 is connected to a local computer system 60 for data processing. The operating room modular devices 1a-1n may be connected to the modular communication hub 53. The network hub 57 and / or the network switch 59 are connected to the network router 61 so that devices 1a-1n can connect to the cloud 54 or the local computer system 60. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 via a wired channel or a wireless channel. The surgical data network environment 51 may be employed, as described herein, to provide an augmented reality interactive experience during a surgical procedure, in particular to provide augmented images of the surgical field to one or more remote displays 58.

[0032] Figure 5 shows a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 which may include a remote server 63. In one embodiment, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to a plurality of surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 connected to a computer system 60.

[0033] Returning to Figure 5, the modular control tower 23 is connected to an imaging module 38 connected to an endoscope 98, a generator module 27 connected to an energy device 99, a fume exhaust module 76, a suction / irrigation module 78, a communication module 13, a processor module 15, a storage array 16, and optionally smart devices / instruments 21 and sensor modules 29 connected to a display 39. Surgical site devices are connected to cloud computing resources such as a server 63, data storage 55, and a display 58 via the modular control tower 23. The robot hub 72 may also be connected to the modular control tower 23, as well as the server 63, data storage 55, and display 58. In particular, the devices / instruments 21 and the visualization system 58 may be connected to the modular control tower 23 via wired or wireless communication standards or protocols as described herein. The modular control tower 23 may be connected to a hub display 65 (e.g., a monitor, screen) to display received augmented images, including overlaid virtual objects on the real field, received from the imaging module 38, device / instrument display 39, and / or other visualization systems 58. The hub display 65 may also display data received from devices connected to the modular control tower 23, along with the images and overlay images.

[0034] Figure 6 shows a surgical hub 56 including multiple modules connected to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, such as a network connectivity device, and a computer system 60 for local processing, visualization, and imaging of augmented surgical information. The modular communication hub 53 is connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53, and data associated with the modules may be transferred to the computer system 60, cloud computing resources, or both. Each of the network hubs / switches 57 / 59 within the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs / switches 57, 59 are connected to the processor 31 to provide communication connectivity to cloud computing resources and local displays 67. Communication to the cloud 54 can be done via either a wired communication channel or a wireless communication channel.

[0035] The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is connected via a system bus to a communication module 41, storage 45, memory 46, non-volatile memory 47, and an input / output interface 48. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using various available bus architectures.

[0036] The processor 31 may include an augmented reality modeler (e.g., as shown in Figure 10) and may be implemented as a single-core or multi-core processor, such as one known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0037] System memory includes volatile and non-volatile memory. The Basic Input / Output System (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory may include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0038] The computer system 60 also includes removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0039] In various embodiments, the computer system 60 in Figure 6, the imaging module 38 in Figures 4 to 6, and / or the visualization system 58, and / or the processor module 15 may include an image processor, an image processing engine, a graphics processing unit (GPU), a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0040] Figure 7 shows an augmented reality system 263 that includes an intermediate signal coupler 64 located in the communication path between the imaging module 38 and the surgical hub display 67. The signal coupler 64 combines audio and / or image data received from the imaging module 38 and / or the AR device 66. The surgical hub 56 receives the combined data from the coupler 64, overlays the provided data onto the display 67, and displays the overlaid data. The imaging device 68 may be a digital video camera, and the audio device 69 may be a microphone. The signal coupler 64 may include a wireless head-up display adapter for coupling to the AR device 66 located in the communication path of the display 67 to a console that enables the surgical hub 56 to overlay data onto the display 67.

[0041] Figure 8 shows an augmented reality (AR) system including an intermediate signal coupler positioned in the communication path between the imaging module and the surgical hub display. Figure 8 shows an AR device 66 worn by the surgeon 73 to communicate data to the surgical hub 56. Peripheral information of the AR device 66 does not include active images. Rather, peripheral information includes only signals that do not have the same requirements for device settings or refresh rate. The interaction may extend the surgeon 73's information based on links with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures and, for example, ask whether an instrument is touching a nerve, blood vessel, or adhesion. The AR device 66 may include processing in the surgical hub 56 used to provide preoperative scan data, optical views, tissue examination characteristics acquired throughout the procedure, and / or answers. The surgeon 73 may write notes on the AR device 66 so that they are stored in the hub storage 45 along with patient data for later use in reporting or follow-up.

[0042] The AR device 66, worn by the surgeon 73, links to the surgical hub 56 using auditory and visual information to avoid the need for overlays, and allows customization of information displayed around the periphery of the field of view. The AR device 66 provides signals from devices (e.g., instruments) and answers queries regarding location information linked with video to identify device settings or quadrants or locations. The AR device 66 has voice control and voice feedback from the AR device 66. The AR device 66 can interact with other systems in the operating room and can have available feedback and interaction wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture start commands and queries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities, including voice, visual, or haptic touch.

[0043] Figure 9 shows a surgeon 73 and a patient 74 wearing AR devices 66, and may include a camera 96 ​​in the operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects overlaid on a real-time image of the surgical field to the surgeon 73 through an augmented reality display 89 or through a hub-connected display 67. The real-time image may include parts of surgical instruments 77. The virtual objects may not be visible to others in the operating room 75 (e.g., surgical assistants or nurses), but they may also wear AR devices 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects, or may be able to see the virtual objects in augmented reality shared with the surgeon 73, or may be able to see modified versions of the virtual objects (e.g., according to customization specific to the surgeon 73), or may see different virtual objects.

[0044] Virtual objects and / or data may be configured to appear on a portion of the surgical instrument 77 or within the surgical field captured by the imaging module 38, the imaging device 68 during minimally invasive surgical procedures, and / or the camera 96 ​​during incisional surgical procedures. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides live images of the surgical area during minimally invasive surgical procedures. The AR system may present virtual objects fixed to real objects regardless of the viewpoint of one or more viewers of the AR system (e.g., the surgeon 73). For example, virtual objects may be visible to viewers of the AR system inside the operating room 75, but not visible to viewers of the AR system outside the operating room 75. Virtual objects may be displayed to viewers outside the operating room 75 when a viewer enters the operating room 75. Augmented images may be displayed on the surgical hub display 67 or the augmented reality display 89.

[0045] The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each user's eye). The screens may allow light to pass through them so that aspects of the real environment are visible while virtual objects are being displayed. Virtual objects may become visible to the surgeon 73 by projecting light. Virtual objects may appear to have some degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

[0046] An AR system may be visible to one or more viewers and may include differences between views available to one or more viewers, while maintaining some common aspects between views. For example, a heads-up display may change between two views, but virtual objects and / or data may be fixed to real objects or areas in both views. Aspects such as the color, lighting, or other changes of an object may occur between views without changing the fixed position of at least one virtual object.

[0047] Users can view virtual objects and / or data presented within the AR system as opaque or with a certain level of transparency. For example, a user can interact with a virtual object by moving it from a first position to a second position. For instance, a user may move an object with their own hand. This may be done virtually in the AR system by determining that the hand has moved to a position adjacent to or adjacent to the object (using one or more cameras, which may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controlled to move) and moving the object accordingly. The virtual form may include a virtual representation of a real-world object, or it may include visual effects such as lighting effects. The AR system may include rules to govern the behavior of the virtual object, such as exposing the virtual object to gravity or friction, or it may include other predefined rules that negate real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device 66 may include a camera 79 (which should not be confused with a separate camera 96). The AR device camera 79 or camera 96 ​​may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. The AR device 66 may project virtual items onto a representation of the real environment that the user can see.

[0048] The AR device 66 may be used, for example, in an operating room 75 during a surgical procedure performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects, such as virtual objects during the surgical procedure, to extend the surgeon's vision. The surgeon 73 may view the virtual objects using the AR device 66, a remote controller for the AR device 66, or interact with the virtual objects by using his hands to “interact” with the virtual objects or gestures recognized by the camera 79 of the AR device 66, for example. The virtual objects can extend surgical tools, such as surgical instruments 77. For example, the virtual object may appear to be connected to the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66), or to remain at a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide the surgical instrument 77 and may appear to be fixed to the patient 74. In certain examples, the virtual object may react to the movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

[0049] The augmented reality display system imaging device 38 captures real images of the surgical area during the surgical procedure. The augmented reality displays 89 and 67 present an overlay of the operating modes of the surgical instrument 77 onto the real images of the surgical area. The surgical instrument 77 includes a communication circuit 231 for communicating operating mode and functional data from the surgical instrument 77 to the AR device 66 via a communication circuit 233 on the AR device 66. The surgical instrument 77 and the AR device 66 are shown in RF wireless communication between circuits 231 and 233, as indicated by arrows B and C, but other communication technologies (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operating modes of the surgical instrument 77 that are actively visualized. The overlay combines the modes of tissue interaction in the surgical area with functional data from the surgical instrument 77. The processor portion of the AR device 66 is configured to receive operating mode and functional data from the surgical instrument 77, determine overlays related to the operation of the surgical instrument 77, and combine the tissue characteristics within the surgical area with the functional data from the surgical instrument 77. The augmented images display alerts regarding device performance considerations, non-conformity use, and incomplete capture. Non-conformity use includes out-of-range tissue conditions and tissue improperly balanced within the jaws of the end effector. Additional augmented images provide displays of incidental events, including tissue tension and foreign body detection. Other augmented images display device status overlays and instrument displays.

[0050] Figure 10 shows a system 83 for augmenting images of the surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes one aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, memory 87, an AR display 89, and a camera 79. The AR device 66 may also include a sensor 90, a speaker 91, and / or a haptic controller 92. The database 93 may include image storage 94 or preoperative planning storage 95.

[0051] The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive images of instruments in the surgical field from a camera 79 or sensor 90, etc., and create an augmented reality environment that fits within the displayed image of the surgical field. In another example, physical objects and / or data may be overlaid on the surgical field and / or surgical instrument images, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of virtual objects and / or data within the augmented reality environment. For example, the augmented reality modeler 86 may use or detect instruments at the patient's surgical site and present virtual objects and / or data on the surgical instruments, and / or images of the surgical site in the surgical field captured by the camera 79. The AR display 89 may display the AR environment overlaid on the real environment. The display 89 can use the AR device 66, which is located in a fixed position, etc., within the AR environment, to show virtual objects and / or data.

[0052] The AR device 66 may include sensors 90 such as infrared sensors. The camera 79 or sensor 90 may be used to detect movements such as gestures by a surgeon or other user, which may be interpreted by the processor 85 as attempted or intended interactions by the user with a virtual target. The processor 85 can identify objects in the real environment by processing information received using the camera 79, for example. In other embodiments, sensor 90 may be a tactile sensor, audible sensor, chemical sensor, or thermal sensor to generate corresponding signals that can be combined with various data feeds to create an augmented environment. Sensor 90 may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye-tracking sensors, and voice command recognition capabilities.

[0053] The AR display 89 may, for example, during a surgical procedure, allow the surgical field to be viewed through the AR display 89, while presenting virtual features within the surgical field that correspond to physical features hidden by the patient's anatomical features. The virtual features may have a virtual position or orientation that corresponds to a first physical position or orientation of the physical features. In one example, the virtual position or orientation of the virtual features may include an offset from the first physical position or orientation of the physical features. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical features, and so on.

[0054] In one example, the AR device 66 may be an individual AR device. In one embodiment, the AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft in Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometer, gyroscope, magnetometer), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition capabilities. It provides a high-resolution, improved field of view by using mirrors to orient waveguides in front of the wearer's eyes. The image can be magnified by changing the angle of the mirrors. It also provides eye tracking to recognize the user and adjust the lens width for a particular user.

[0055] In another example, AR device 66 could be the Snapchat Spectacles 3 AR device. This AR device offers the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras for capturing 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to construct a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synchronized to an external display device.

[0056] In yet another example, AR device 66 could be Google's Glass 2 AR device. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

[0057] In another example, AR device 66 could be an Echo Frames AR device from Amazon. This AR device does not have a camera / display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

[0058] In yet another example, AR device 66 could be the Focals AR device by North (Google). This AR device provides a notification pusher / smartwatch analog, inertial measurement, screen overlays for information (weather, calendar, messages), and voice control (Alexa) integration. This AR device also provides basic head-up display functionality.

[0059] In another example, AR device 66 could be an Nreal AR device. This AR device includes spatial sound, two ambient cameras, a photographic camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and proximity sensor functions. Nebula projects application information onto the lens.

[0060] In various other examples, the AR device 66 may be any one of the following commercially available AR devices, namely Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, or EyeTap, which create light collinear with the ambient light directly onto the retina. A beam splitter makes the same visible light available to a computer, for example, to process and overlay information. The AR visualization system may include a HUD, contact lenses, glasses, virtual reality (VR) headset, virtual retinal display, intraoperative display, and / or smart contact lenses (bionic lenses).

[0061] The multi-user interface for the AR device 66 includes a virtual retinal display such as a raster display that draws directly onto the retina rather than on a screen in front of the eyes, a smart TV, a smartphone, and / or a spatial display such as the Sony Spatial Display System.

[0062] Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, the Apple Polycam app and Ubiquity 6 (Mirrorworld using the Display.land app), which allow users to scan and acquire 3D images of the real world (to create 3D models). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, and Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, and Magic Leap.

[0063] Situational awareness is the ability of several embodiments of a surgical system to determine or infer information related to a surgical procedure from data received from a database and / or instruments. This information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. Based on contextual information relating to a surgical procedure, the surgical system can be improved, for example, by controlling modular devices connected to it (e.g., robotic arms and / or robotic surgical tools) and providing contextualized information or suggestions to the surgeon during the course of the surgical procedure.

[0064] Figure 11 shows a timeline of a situation-aware surgical procedure. Figure 11 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each step of the surgical procedure. Timeline 5200 shows typical steps that nurses, surgeons, and other healthcare workers might take during a lung segmentectomy procedure, which begins with setting up the operating room and ends with transferring the patient to the postoperative recovery room. Throughout the course of the surgical procedure, the situation-aware surgical hub 5104 receives data from data source 5126, including data generated each time healthcare workers use the modular device 5102 paired with the surgical hub 5104. By receiving this data from the paired modular device 5102 and other data sources 5126, the surgical hub 5104 can continuously derive estimations (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub 5104 can, for example, record data relating to a procedure to generate a report, verify the steps being taken by a healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as described above.

[0065] In the first step 5202, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery.

[0066] In the second 5204, staff scan incoming medical supplies for a procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies used in various types of procedures to confirm that the mixture of supplies corresponds to a thoracic procedure. Furthermore, the surgical hub 5104 can also determine that the procedure is not a wedge resection (because the incoming supplies either do not contain specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection).

[0067] In the third 5206, a healthcare worker scans the patient band via a scanner 5128 that is communicably connected to a surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

[0068] In the fourth part of 5208, a medical professional turns on the assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they include a fume exhauster, an air inlet, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104 located within a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on the combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team. Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing.

[0069] In step 5210, the staff attaches the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 confirms that the patient is in the operating room.

[0070] In step 6, 5212, medical personnel induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from modular devices 5102 and / or patient monitoring devices 5124, including, for example, EKG data, blood pressure data, ventilator data or a combination thereof. Once step 6, 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion commences.

[0071] In section 7 of 5214, the lung of the patient being operated on collapses (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has begun and determine that collapsing the lung is the first surgical step in this particular procedure.

[0072] In step 8, 5216, a medical imaging device 5108 (e.g., a scope) is inserted, and video footage from the medical imaging device is initiated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or real-time live-streaming video) through its connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resection has not been taken into consideration by the surgical hub 5104 based on the data received in step 2 of the procedure, 5204). Using data from the medical imaging device 124 (Figure 2), contextual information regarding the type of procedure being performed can be determined in various ways, for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number of medical imaging devices being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used.

[0073] For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. The situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to a surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the techniques used for that specific type of surgical procedure.

[0074] In step 9, 5218, the surgical team initiates the incision step of the procedure. The surgical hub 5104 receives data from the RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the procedure steps described above) corresponds to the incision step.

[0075] In the tenth step 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 receives data from the surgical stapling and cutting instruments indicating that instruments are being fired, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps in the read-out process.

[0076] In the eleventh step 5222, the segmental resection portion of the procedure is performed. The surgical hub 5104 estimates that the surgeon has transversely incised parenchymal tissue, based on data from surgical instruments, including data from a staple cartridge. Cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Cartridge data may indicate the type of tissue being stapled and / or transversely incised, for different types of staples used for different types of tissue. The type of staples fired is used for parenchymal tissue or other similar tissue types, and the surgical hub 5104 can estimate that a segmental resection procedure has been performed.

[0077] Next, in the twelfth step 5224, the nodule incision step is performed. Based on the data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby enabling the surgical hub 5104 to make this inference. Note that the surgeon will periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once the twelfth step 5224 is completed, the incision is closed and the postoperative portion of the procedure begins.

[0078] In the 13th step 5226, the patient is de-anesthetized. The surgical hub 5104 can estimate that the patient is waking up from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase).

[0079] Finally, in the 14th step 5228, the healthcare worker removes the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 can infer that the patient has been transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. Based on the data received from various data sources 5126 that are communicably connected to the surgical hub 5104, the surgical hub 5104 can determine or infer when each step of a given surgical procedure is occurring.

[0080] As shown in the first step 5202 of the timeline 5200 in Figure 11, in addition to estimating the type of surgical procedure to be performed using patient data from the EMR database(s), the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0081] Mixed elements and actions occurring on and off screen are directly visualized using rendered elements. Having described common implementations of various surgical systems, surgical hubs, communication systems, augmentation systems, and augmented reality devices disclosed herein, such as surgical systems 1, 2, 50, 52, surgical hubs 6, 56, 5104, communication system 63, visualization system 8, augmentation system 83, imaging devices 24, 96, and AR devices 66, 84, this disclosure now describes various other embeddings of systems, hubs, and devices. For brevity, the various details and implementations of systems, hubs, and devices described in the following sections are similar to the various systems, hubs, and devices described above and will not be repeated herein. Any aspect of the systems, hubs, and devices described below can be incorporated into and / or implemented by the above systems, hubs, and devices.

[0082] As described above, augmented reality display devices and other types of display devices can be used to provide information overlays to operating room (OR) personnel during surgical procedures. In some embodiments, these overlays may include information related to the steps of the procedure being performed by the OR personnel. Therefore, the information displayed by the overlays may need to be based on the surgical instruments being used by the personnel or the area of ​​the surgical field in which they are working. However, during a surgical procedure, there are often multiple OR personnel interacting with a wide variety of surgical instruments and other objects. Furthermore, surgeons, nurses, and assistants may all be working in and around the surgical field at various times. Therefore, each personnel may move around the operating room, operate multiple surgical instruments, hand instruments to each other, and set instruments aside when not in use. Given the constantly changing situation in the OR, it can be difficult for a surgical system to track and consolidate the information that needs to be displayed to various personnel on various devices throughout the surgical procedure. Therefore, there is a need for devices, systems, and methods for tracking multiple users and objects in the OR so that relevant information can be appropriately displayed by various augmented reality and other display devices.

[0083] Furthermore, at various points during a surgical procedure, personnel, instruments, and other objects may move in and out of the field of view of various imaging devices in a surgical system, such as imaging devices configured to capture images of the surgical field. As a result, personnel relying on augmented reality and other display devices displaying captured images of the surgical field may not be able to see some of the instruments being actively used. Therefore, personnel may not be able to accurately perceive important attributes of the instruments. For example, a surgeon performing a transverse incision using an endocutter may not be able to see part of the endocutter as it passes outside the field of view of the endoscope. Because the field of view is obstructed, the surgeon may not be able to perceive the range of articulation of the end effector of the endocutter. Or, in another example, the surgeon may not be able to perceive the position of the end effector. Therefore, the surgeon may have difficulty performing the transaction accurately. Thus, there is a need for devices, systems, and methods for tracking the attributes of surgical instruments outside the field of view of the imaging device and displaying the tracked attributes using overlays on augmented reality devices and other display devices.

[0084] Furthermore, at various points during a surgical procedure, surgical instruments and other objects may be outside the field of view of various imaging devices. Therefore, personnel relying on augmented reality and other display devices may not be able to perceive potential interactions between surgical instruments and other objects outside the field of view. For example, a surgeon may be using an energy device for a step in a surgical procedure. However, while viewing a display device showing a live image of the surgical field captured by the endoscope, the surgeon may not be able to perceive that the energy device is in close proximity to a metal instrument outside the field of view of the endoscope. Therefore, there is a risk that the surgeon may activate the energy device in close proximity to the metal instrument, causing the energy device to malfunction (e.g., an electric arc). As another example, a surgeon attempting to perform a procedure using a circular stapler may be able to see the device deck of the stapler within the field of view of the imaging device, but may not be able to see the anvil which is outside the field of view. Therefore, the surgeon may have difficulty guiding and manipulating the tissue to optimize the attachment of the device deck and anvil. As yet another example, a surgeon operating a device may not be able to perceive potential collisions or unintended interactions between surgical instruments and objects outside the field of view of the imaging device. Therefore, there is a need for apparatus, systems, and methods for predicting interactions between surgical instruments and objects outside the field of view of an imaging device, and for displaying attributes of surgical instruments related to potential interactions.

[0085] Real-time object location tracking In various embodiments, devices, systems, and methods for tracking multiple users and objects within an OR are disclosed herein. In some embodiments, these devices, systems, and methods for tracking multiple users and objects may be employed to ensure that relevant information relating to the tracked object can be displayed to a specific user(s) using various augmented reality and other display devices disclosed herein.

[0086] Figure 12 shows an example of a surgical system 15000 for tracking the location of an object in an OR, according to some non-limiting aspects of the present disclosure. The surgical system 15000 may include a tracking system 15006 and a surgical hub 15002 that communicates with at least one AR device 66. In some embodiments, the tracking system 15006 may include a visualization system 15008. In many embodiments, the surgical system 15000, the surgical hub 15002, and the visualization system 15008 may be similar to any of the surgical systems, surgical hubs, and visualization systems described above (e.g., surgical systems 1, 2, 50, 52; surgical hubs 6, 56, 5104; visualization systems 8, 58). In some non-limiting aspects of Figure 12, the tracking system 15006 communicates with the surgical hub 15002. In other embodiments, the surgical hub may include a module comprising the tracking system 15006. The surgical hub 15002 may include an operating room mapping module 15004 configured to map the location and / or status of various objects within the OR based on data received from the tracking system 15006, as will be described in more detail below.

[0087] The tracking system 15006 can be configured to track the position and / or other attributes of various objects within the OR based on one or more different types of tracking methods. In one embodiment, the tracking system 15006 (and / or the visualization system 15008) may include one or more imaging devices 15010. In many embodiments, the imaging devices 15010 may be similar to the imaging devices 24, 96, AR device 66 and / or other imaging sensors described above with respect to the visualization system 8. Thus, the imaging devices 15010 may include cameras and other types of visible and invisible sensors for capturing images within the OR or otherwise tracking objects. For example, the imaging devices 15010 may employ visual, infrared, and / or other higher wavelength image recognition techniques to establish the movement and location of objects within the OR. The imaging devices 15010 may be installed in multiple locations throughout the operating room with overlapping fields of view so that images of objects within the OR can be captured and tracked from multiple angles. Furthermore, multiple imaging devices 15010 may be implemented so that when an object in the OR leaves the field of view of a first imaging device 15010 (e.g., a first camera 15010), that object can be tracked by at least a second imaging device 15010 (e.g., a second camera 15010).

[0088] In another embodiment, the tracking system 15006 may include one or more structured light sensors 15012 (e.g., structured light scanners) configured to track objects within the OR. The structured light sensors 15012 may be configured to project a defined pattern of light from multiple angles, for example, to triangulate the position of an object within the OR based on the distortion of the pattern caused by the object. The structured light sensors 15012 may be similar to and / or include devices such as Microsoft Kinect, Intel F200, Intel R200, and / or Occipital Structure.

[0089] In another embodiment, the tracking system 15006 may include one or more LiDAR (light detection and ranging) sensors 15014 configured to track objects within the OR. In yet another embodiment, sensors using LiDAR-like techniques may be employed by the tracking system 15006 to track objects within the OR.

[0090] In another embodiment, the tracking system 15006 may include one or more floor sensors 15016 configured to track objects within the OR. The floor sensors 15016 may include weight sensors. In one embodiment, the tracking system may include an array of floor sensors 15016 configured to determine where equipment is located within the OR. For example, referring here to Figures 2 and 12, the floor sensors 15016 may be configured to determine the location and / or position of an operating table 14, a surgical console 18, a robot hub 22, a side cart 20, etc. In another embodiment, the floor sensors 15016 may be configured to monitor the location and / or position of personnel within the OR.

[0091] The data generated by the floor sensor 15006 can be processed by the surgical hub 15002 to make decisions related to various aspects of the surgical procedure. For example, the weight measured by the floor sensor 15006 can be used to determine whether a device is placed on a side cart or other parts of the OR equipment (for example, by recognizing the change in weight when the device is placed). In another example, the floor sensor 15006 can be used to detect fatigue of OR personnel based on their movement (e.g., swaying, weight distribution). In yet another example, the floor sensor can be used to track a patient's weight during surgery. The patient's weight can be verified by the surgical hub 15002 throughout the procedure for various reasons, such as ensuring that the dosage of medication administered to the patient is within the acceptable range for that patient's weight, so that it is tracked by the tracking system 15006. In yet another example, the floor sensor 15016 can be used to track medical waste, devices, equipment, etc. that fall to the floor during the procedure.

[0092] Referring further to Figure 12, in another embodiment, the tracking system 15006 may include one or more acoustic sensors 15018 configured to monitor the position, location, and / or movement of objects within the OR. For example, the acoustic sensors 15018 may employ voice beacon technology, phase-coherent tracking, and / or time-of-flight triangulation to establish the position and movement of objects within the OR.

[0093] In another embodiment, the tracking system 15006 may include one or more reference markers 15020. In one embodiment, the reference marker 15020 may be any type of marker configured to assist in tracking the location, position, and / or movement of an object relative to the field of view of the imaging device 15010 and / or to location, position, and / or movement data tracked by any of the other devices / sensors of the tracking system 15006. For example, the reference marker(s) 15020 may include an RFID chip configured to track the location and / or position of an object to which an RFID (radio frequency identification) chip is attached. Thus, in some embodiments, the reference marker(s) 15020 may be placed in and / or on surgical devices, operating room equipment, objects worn by OR personnel, or any other object that can be tracked by the tracking system 15006. In some embodiments, tracking of the reference marker 15020 by the tracking system 15006 can be triggered to begin based on the occurrence of events such as, for example, the removal of an object (e.g., a device) containing the reference marker 15020 from its packaging, the insertion of a battery into an object (e.g., a device) containing the reference marker 15020, and / or when the object containing the reference marker 15020 enters an OR. The reference marker 15020 can be used to assist in the generation of augmented reality overlays, as will be discussed in more detail below.

[0094] In another embodiment, the tracking system 15006 may include one or more user / device sensors 15022 configured to identify and monitor the location, position, and / or movement of OR personnel and / or devices within the OR. In one embodiment, the user / device sensor 15022 may be included in a device or apparatus worn by OR personnel. The user / device sensor 15022 may include, for example, an accelerometer, a gyroscope, and / or a magnetometer for tracking the three-dimensional movement of OR personnel and / or devices. In another embodiment, the user / device sensor 15022 may include an RFID bracelet worn by OR personnel. In one embodiment, data from the user / device sensor 15022 may be used by the surgical hub 15002 and / or the tracking system 15006 to associate a device (e.g., a surgical instrument) with a specific user within the OR at a given time during a surgical procedure. For example, the tracking system 15006 and / or the surgical hub 15002 may be configured to track the movement of OR personnel and devices separately using multiple user / device sensors 15022. When the tracking system 15006 detects that a user / device sensor 15022 worn by an OR personnel is in proximity to a user / device sensor 15022 associated with a surgical instrument, the surgical hub 15006 can identify that the OR personnel is associated with (e.g., linked to, using) the surgical instrument. Based on the identified association between personnel and instruments, the surgical hub 15002 can trigger the generation of augmented reality overlays specific to the personnel and / or surgical instruments, as will be described in more detail below with respect to Figures 12-14. In another embodiment, the user / device sensors 15022 can be used to identify instruments and / or devices. The user / device sensors 15022 may include RFID tags to identify specific types of devices being used during a procedure. For example, the user / device sensor 15022 could be an RFID tag on the trocar to identify the type of trocar being used.

[0095] In another embodiment, the tracking system 15006 may include one or more GPS (Global Positioning System) sensors 15024 that are tracked using GPS (e.g., satellite) tracking technology to monitor the position, location, and / or movement of objects within the OR. The tracking system 15006 in Figure 12 is shown to implement tracking technology including imaging devices 15010, structured light sensors 15012, LiDAR sensors 15014, floor sensors 15016, acoustic sensors 15018, reference markers 15020, user / device sensors 15022, and GPS 15024, but it should be noted that the tracking system 15006 can be configured to use any combination of these technologies (e.g., including only some of these technologies). Furthermore, in some embodiments, other tracking techniques configured to track the location, position, movement, and / or other attributes of objects within the OR may be implemented by the tracking system 15006.

[0096] Figure 13 is a schematic side view of exemplary implementations of a tracking system 15006 in an operating room (OR) 15050 according to some non-limiting aspects of the present disclosure. The OR 15050 may be any part of the OR. For example, in some embodiments, Figure 13 may show an overall side view of the OR 15050. In other embodiments, Figure 13 may show a side section of the surgical field. The tracking system 15006 may include a first tracking device 15054 and a second tracking device 15056. The first tracking device 15056 and / or the second tracking device 15054 may be implemented using any combination of the tracking techniques mentioned above with respect to Figure 12 (e.g., imaging device(s) 15010, structured optical sensor(s) 15012, LIDAR sensor(s) 15014, acoustic sensor(s) 15018, etc.). The first tracking device 15056 and the second tracking device 15054 may implement the same tracking technology or different tracking technologies.

[0097] Referring primarily to Figure 13, and also to Figure 12, in some embodiments, the first tracking device 15054 and / or the second tracking device 15056 may be configured to track the first portion 15058 and the second portion 15060 of the target object 15062. The target object 15062 may be an object or area such as an object in the surgical field, an area within a sterile barrier, a patient, tissue, OR equipment, surgical devices, OR personnel, etc. The first tracking device 15054 may be capable of directly tracking the first portion 15058 and the second portion 15060 of the target object 15062 (15055A, 15055B). For example, the first tracking device 15054 may be a camera (e.g., imaging device 15010) in which the first portion 15058 and the second portion 15060 of the target object 15062 are directly within the camera's field of view (15055A, 15055B). The second tracking device 15056 can directly track the first portion 15058 of the target object 15062 (15057A). However, the second tracking device 15056 may not be able to track the second portion 15060 (15057B). For example, the second tracking device may be a camera (e.g., imaging device 15010), and the second portion 15060 may be outside the camera's field of view. This may be because an interfering object 15064 (e.g., OR personnel, surgical instruments, tissue, etc.) is blocking the tracking 15057B of the second portion 15060 of the target object 15062. Although the interfering object 15064 blocks the tracking of the second portion 15060 of the target object 15062 by the second tracking device 15056 (15057B), the tracking system 15006 can still track the second portion 15060 because the first tracking device 15054 directly tracks the second portion 15060 (15055B).Therefore, the tracking system 15060 can be configured to have tracking devices that track overlapping tracking areas (e.g., multiple imaging devices / systems with overlapping fields of view) so that the target object can be tracked when the target object or a part of the target object is outside the field of view of one of the tracking devices. Thus, the surgical hub 15002 can display information related to the target object (e.g., images, overlays, notifications, etc.) on the display device even when the target object or a part of it is outside the field of view of one of the tracking devices.

[0098] Referring primarily to Figure 13, and also to Figure 12, in other embodiments, the first tracking device 15054 and / or the second tracking device 15056 may include tracking devices configured to directly track (15055A, 15055B, 15057A) and / or indirectly track (15057C, 15057D) the first portion 15058 and the second portion 15060 of the target object 15062. For example, the second tracking device 15056 may be a device implementing reflective tracking technology (e.g., acoustic sensor(s) 15018), thereby allowing the second tracking device 15056 to indirectly track (15057C, 15057D) the target object 15062 (e.g., based on reflections using object 15066) when the interfering object 15064 is prohibited from direct tracking (15057B). Therefore, the tracking system 15060 can be configured to have tracking devices with overlapping tracking areas (e.g., areas tracked using non-image technology, reflective technology, voice beacons, GPS, RFID, etc.) in order to track the location, position, movement, and / or other attributes of a target object.

[0099] Figure 14 shows a schematic plan view of an exemplary operating room map 15070 generated by the operating room mapping module 15004 of Figure 12, according to at least one non-limiting aspect of the present disclosure. Referring primarily to Figure 14, and also to Figures 12-15, the tracking system 15006 can transmit tracking data generated using any combination of tracking techniques (e.g., imaging device(s) 15010, structured optical sensor(s) 15012, LIDAR sensor(s) 15014, floor sensor(s) 15016, acoustic sensor(s) 15018, reference marker(s) 15020, user / device sensor(s) 15022, and GPS(s) 15024) to the operating room mapping module 15004 of the surgical hub 15002. Based on the tracking data, the operating room mapping module 15004 can generate an operating room map 15070 of the operating room 15060. Map 15070 may include information about the location, position, movement, and / or other attributes of multiple objects (e.g., 15072A-15072L) within the operating room 15060. For example, objects 15072A-15072L in map 15070 may correspond to various devices, equipment, OR personnel present in the operating room. Map 15070 can be updated in real time based on tracking data from the tracking system 15006. In some embodiments, map 15070 may be displayed by any of the display devices disclosed herein. In other embodiments, the surgical hub 15002 may determine the proximity and / or interaction of objects based on map 15070.

[0100] Multi-user tracking and information association Referring to Figures 12-14, in other embodiments, augmented reality overlays and other notifications and alerts can be generated based on the location, position, and / or motion of objects (e.g., 15072A-15072L) as determined by the operating room mapping module 15004. In some embodiments, users and devices can be tracked using a tracking system 15006 based on both user / device sensors 15022 and other sensing technologies (e.g., imaging device(s) 15010, structured light sensor(s) 15012, LIDAR sensor(s) 15014, floor sensor(s) 15016, acoustic sensor(s) 15018, GPS 15024, etc.). In some embodiments, as described above, user and device sensor data can be used by the surgical hub 15002 (e.g., the operating room mapping module 15004) and / or the tracking system 15006 to associate a device (e.g., a surgical instrument) with a specific user in the OR at a given time during a surgical procedure. In this embodiment, both active (wearable) and passive (camera) tracking methods can be used by the tracking system 15006 and the operating room mapping module 15004 to map the location of devices and personnel within the OR suite (15070). For example, a user (surgeon and other OR personnel) may wear gloves containing user / device sensors 15022. In one embodiment, the surgical hub 15002 can be configured to identify that the gloves are linked to the user's right and left hands.

[0101] In some embodiments, the operating room mapping module 15004 can be configured to associate users with specific locations within the operating room map 15070. For example, the operating room map 15070 may divide the OR 15060 into specific areas. Based on data from the tracking system 15006, the operating room mapping module 15004 can identify users who have preferred and / or control over a device. When a user swaps a device (e.g., transfers physical control of the device to a different user), a sensor (e.g., a device / user sensor 15022) can detect the swap, thereby allowing the surgical hub 15002 to identify which user is associated with the device. For example, gloves worn by a user may be configured with sensors that track finger pressure and position. Based on the glove sensor data, device sensor data, and / or data from the imaging device 15010, the surgical hub 15002 (e.g., the operating room mapping module 15004) can determine who is controlling the device and calculate the likelihood of who is guiding the device.

[0102] In various embodiments, the surgical hub 15002 can display notifications, alerts, and / or overlays on any of the displays and / or AR devices described herein (displays 7, 9, 19; AR devices 66, 84) based on data from the tracking system 15006, the operating room mapping module 15004, and / or the surgical hub 15002. In one embodiment, a notification can be displayed to one or more users based on the determination by the operating room mapping module 15004 that a user is controlling a surgical instrument. For example, an AR device worn by a surgeon can display a notification indicating that the surgeon has taken control of a surgical instrument handed to the surgeon by another OR staff member. As another example, both a first AR device worn by a surgeon and a second AR device worn by an OR staff member can display a notification indicating that the surgeon has obtained control of a surgical instrument from the OR staff member. In some embodiments, whether or not a particular user's AR device displays a notification can be based on a priority level related to data tracked by the tracking system 15006 and / or information determined by the surgical hub 15002.

[0103] In one embodiment, a surgical device may be intended to be used simultaneously by multiple users. In this embodiment, parts (e.g., sections) of the surgical device can be individually tracked by a tracking system 15006. For example, a circular stapling device may include a device part having a retractable trocar controlled by an adjustable knob. The circular stapling device may also include a mountable anvil part. Different parts of the circular stapling device are controlled by different users and can be tracked separately by the tracking system 15006. Based on data from the tracking system 15006, a display device (e.g., an AR device) associated with each user can be configured to display different overlays based on the part of the device that the user controls. For example, a display associated with a user controlling an adjustable trocar can display an overlay based on the dial pressure when the user adjusts the knob. Displays associated with both users may show the state in which the anvil is mounted on the trocar.

[0104] In some embodiments, the user may not be wearing a trackable sensor (e.g., user / device sensor 15022). The tracking system 15006 can be configured to track the actions of the user and / or user-controlled devices using passive tracking (e.g., using imaging device 15010). For example, a nurse may not be wearing a trackable sensor. The nurse may reload a staple cartridge into the endocutter. Specific cartridge types can be color-coded. Reloading and replacements performed by the nurse can be detected based on the tracking system's camera. Furthermore, notifications based on reloading and replacements detected by the camera can be displayed to the user whom the surgical hub 15002 has determined last used the device (e.g., a surgeon wearing an active tracking device). The notification may include an overlay indicating the type of cartridge reloaded based on the color of the cartridge detected by the tracking system 15006's camera. This tracking can enable the detection of potential errors, such as loading the wrong type of staple cartridge into the endocutter. This tracking can also enable the issuance of warnings (e.g., display notifications) based on these detections. Furthermore, this tracking can provide users with awareness of actions they cannot directly observe.

[0105] Figure 15 is a table 15100 of exemplary tracked object interactions determined by the surgical hub 15002 based on data generated by the tracking system 15006, according to at least one non-limiting aspect of this disclosure. Each tracked object interaction 15102 is associated with a timestamp 15104 and location 15106 of the object(s) interacting. Where applicable, the object interaction is associated with a glove ID 15108 and device ID 15110 based on the user and / or device involved in the object interaction (determined, for example, based on a user / device sensor 15022, imaging device 15010, etc.). If multiple parts of a device are tracked, the object interaction also includes a device part identifier 15112. The table also includes the type of action 15114 determined by the surgical hub 15002 based on the tracked information associated with the object interaction(s). The table also includes tracking sources 15116 (e.g., imaging devices 15010(or more) (cameras); user / device sensors 15022 (wearables)). Thus, based on the data from the tracking system 15006, the surgical hub 15006 can identify various actions that occur during the surgical procedure, such as device handoff and device operation.

[0106] In various embodiments, the information shown in the exemplary table in Figure 15 can be stored in a non-relational database or in a format such as a JSON array, thereby allowing additional information to be associated with the entries (e.g., stored by storage device 5 in Figure 1, storage device 55 in Figure 5, etc.). For example, detected object interactions may be entered into an event log containing various device-related information such as timestamps, device position / rotation, equipment / device ID, and determined actions. In some embodiments, the information can be associated with data sensed by the device, such as tracked force / stress, finger position, and other relevant diagnostic data.

[0107] Off-screen interaction with rendered images created based on predictions In various embodiments, apparatus, systems, and methods are disclosed herein for tracking the attributes of surgical instruments outside the field of view of an imaging device and displaying the tracked attributes using overlays on augmented reality devices and other display devices. Referring again to Figure 12, the surgical system 15000 may include a tracking system 15006 configured to visualize and / or track various objects in the operating room. As described above, in some embodiments, the tracking system 15006 may include a visualization system 15008 and one or more imaging devices 15010. In many embodiments, the visualization system 15008 may be analogous to the visualization systems 8, 58 described herein, and the imaging device(s) 15010 may be analogous in many embodiments to the imaging devices 24, 96, AR device 66, and / or other imaging sensors described herein. Thus, the visualization system 15008 may be configured to capture images of the surgical field (e.g., live feeds) during surgical procedures. For example, the visualization system can capture images of surgical instruments within the surgical field when they are used to perform steps of a surgical procedure. Images captured by the visualization system 15008 can be displayed by one of the display devices disclosed herein, such as an augmented reality (AR) display device, to assist surgical personnel during the surgical procedure.

[0108] Furthermore, as described above, in some embodiments, the tracking system 15006 may utilize reference markers 15020 to track various attributes of surgical devices. Reference markers 15020 can be any type of marker configured to assist in tracking the location, position, and / or movement of an object relative to the field of view of the imaging device 15010 and / or the location, position, and / or movement detected by other sensors / devices of the tracking system 15006. For example, reference markers 15020 may include RFID chips configured to track the location and / or position of objects to which RFID (radio frequency identification) chips are attached. Thus, in some embodiments, reference markers 15020 may be placed in and / or on surgical devices, operating room equipment, objects attached by OR personnel, or any other objects that can be tracked by the tracking system 15006.

[0109] In various embodiments, the surgical hub 15002 can be configured to display on the display of the surgical system 15000 (e.g., AR device 66) an image capture of an object in the surgical field based on the imaging device 15010(or more) overlaid with a graphic representing the attributes of the object determined based on the reference marker 15020. In some embodiments, the reference marker 15020 may be located on / within a surgical instrument. Based on the reference marker(or more), the tracking system 15006 can be configured to identify the type of surgical instrument and / or various other attributes of the surgical instrument associated with the reference marker(or more) 15020.

[0110] In one embodiment, the tracking system 15006 can detect the position and orientation of a reference marker 15020. The reference marker may be positioned on a first part of a surgical instrument. Based on the detected position and orientation of the reference marker 15020, the surgical hub 15002 can determine the position and orientation of a second part of the surgical instrument relative to the image of the surgical field captured by the imaging device 15010. Thus, the surgical hub 15002 can cause the AR device 66 to display a graphic related to the position and orientation of the second part of the surgical instrument, overlaid on the image of the surgical field based on the reference marker 15020. In another embodiment, the second part of the surgical instrument may be outside the field of view of the imaging device 15010. Therefore, the second part of the surgical instrument cannot be observed based solely on the image captured by the imaging device 15010. In this embodiment, the graphic related to the second part of the surgical instrument can be displayed by the AR device 66 as an overlay representing the position and orientation of the second part of the surgical instrument. Therefore, a user viewing the AR device 66 can perceive the position and orientation of the second part of the surgical instrument, even if that part is outside the field of view of the imaging device 15010.

[0111] For example, an end cutter may include a reference marker 15020 on the end cutter's handle. Based on the reference marker 15020, the surgical hub 15002 can determine the position of the end effector or end cutter. A surgeon may be operating the end cutter while viewing images of the surgical field captured by the surgeon using the AR device 66 with the imaging device 15010. The end effector may not be within the field of view of the imaging device 15010. Therefore, to help the surgeon perceive the position and orientation of the end effector, the AR device 66 can display graphics related to the position and orientation of the end effector. These graphics may be, for example, a rendered image of the end effector or a graphic object pointing to the position of the end effector.

[0112] In some embodiments, the tracking system 15006 and / or the surgical hub 15002 can determine various other attributes of an object based on a reference marker 15020. In one embodiment, the reference marker 15020 can be associated with a surgical instrument having a defined range of motion (e.g., defined operating volume and / or area, joint range of motion, rotational range of motion, etc.). Thus, based on the reference marker 15020, the surgical hub 15002 can determine the range of motion of the instrument relative to the image of the surgical field. For example, the endocutter mentioned in the paragraph above may have a joint range of motion and / or rotational range of motion. Thus, based on tracking the reference marker 15020, the surgical hub 15002 can determine the range of motion of the endocutter and display an overlay graphic representing the determined range of motion relative to the image of the surgical field.

[0113] In some embodiments, the tracking system 15006 and / or the surgical hub 15002 can use the reference marker 15020 to verify identification information of objects, such as the identification of surgical instruments and / or their type. For example, as described above, surgical instruments may be communicably connected to a surgical hub (e.g., device / instrument 21 and surgical hub 56 in Figure 5). Based on this connection, the surgical hub can be configured to identify instruments. Identification may include the type of instrument (e.g., 45mm stapler, 60mm stapler, etc.). Thus, the tracking system 15006 and the reference marker 15020 can be used as an alternative and / or redundant means for identifying surgical instruments. In one embodiment, identifying an instrument based on the reference marker 15020 may include determining whether the instrument is active and / or available.

[0114] In some embodiments, the tracking system 15006 and / or the surgical hub 15002 can use a reference marker 15020 as a marker in the OR to provide a zero reference point. In another embodiment, the reference marker 15020 can be positioned at various locations around the OR to provide a reference system that can be used by the tracking system 15006 and / or the surgical hub 15002 to orient other objects tracked by the tracking system 15006. For example, the reference marker 15020 can be placed on a patient / or instrument to determine the relative positions of the patient and the instrument relative to each other. As another example, the reference marker 15020 can be placed on an instrument to determine the proximity and / or relative distance of instruments relative to each other. As yet another example, the reference marker 15020 can be placed on instruments and other equipment in the operating room, such as a table or cart, to determine whether an instrument is positioned on a table.

[0115] In some embodiments, the tracking system 15006 and / or the surgical hub 15002 can use a reference marker 15020 to detect potential accidents and / or safety concerns related to the movement of an object. Detected potential accidents and / or safety concerns may be displayed as notifications via an AR overlay. In one embodiment, the reference marker 15020 can be positioned in various locations to designate a 0 reference point and / or a safety zone. The tracking system 15006 can be configured to detect when an object is approaching a safety zone or when an object is outside a safety zone. For example, a surgical procedure may involve the use of a robotic system in conjunction with laparoscopic instruments. The reference marker 15020 can be positioned to designate a safety zone in which the robotic system can safely operate the laparoscopic instruments. The tracking system 15006 and / or the surgical hub 15002 can be configured to identify when the laparoscopic instruments are outside the safety zone, provide a warning to the user, and / or adjust the operation of the robotic system.

[0116] Figures 16A and 16B show an exemplary intraoperative display 15200, in at least one non-limiting aspect of the present disclosure, which includes an image of a surgical instrument 15204 within the surgical field 15202 and graphics 15212, 15214 representing portions 15210 of the surgical instrument 15204 outside the field of view. The intraoperative display 15002 can be displayed by any of the display devices disclosed herein (e.g., AR device 66). Referring mainly to Figures 16A and 16B and also to Figure 12, an image of the surgical field 15202 can be captured by an imaging device 15010. Based on the image of the surgical field 15202 captured by the imaging device 15010, it can be seen that a first portion 15208 of the surgical instrument 15204 (e.g., a gripping portion) is interacting with the tissue 15206. However, the second portion 15210 of the surgical instrument 15204 (e.g., the shaft portion) is outside the field of view of the imaging device 15010 and therefore cannot be seen in the image of the surgical field 15202. The surgical instrument 15204 may include a reference marker 15020 (not shown in Figures 16A and 16B). Based on the reference marker 15020, the tracking system 15006 and / or the surgical hub 15002 can determine the position of the second portion 15210 of the surgical instrument 15204 relative to the surgical field. Thus, the surgical hub 15002 may include graphics 15212, 15214 representing the position of the second portion 15210 in the intraoperative display 15200. In a non-limiting aspect of Figure 16A, graphic 15212 is a rendered image 15212 of the second portion 15210 of the surgical instrument 15204. In a non-limiting embodiment of Figure 16B, graphic 15214 is a graphic object representing the position of a second portion 15210 of a surgical instrument 15204 relative to an image of the surgical field 15202.

[0117] Therefore, the surgical system 15000 can track the attributes of surgical instruments and display the tracked attributes using an AR display device (e.g., AR device 66) and other display devices disclosed herein. OR personnel may rely on augmented reality and other display devices to display images of the surgical field captured by the imaging device. The surgical system 15000 can enable personnel to perceive parts of surgical instruments that may be outside the field of view of the imaging device. Furthermore, the surgical system 15000 can enable OR personnel to perceive more accurately important attributes of instruments that may not be visible based on a single imaging device, such as the range of motion of the instrument and / or the position of the instrument relative to other tracked objects.

[0118] Prediction of interactions and relationships between objects not within the field of view. In various embodiments, apparatus, systems, and methods for predicting interactions of objects outside the field of view of an imaging device and displaying the attributes of the objects based on the predicted interactions are disclosed herein. Referring again to Figure 12, the surgical system 15000 may include a tracking system 15006 configured to visualize and / or track various objects in the operating room. As described above, in some embodiments, the tracking system 15006 may include a visualization system 15008 and one or more imaging devices 15010. In many embodiments, the visualization system 15008 may be similar to the visualization systems 8, 58 described above, and the imaging device(s) 15010 may, in many embodiments, be similar to the imaging devices 24, 96, AR device 66, and / or other imaging sensors described above. Thus, the visualization system 15008 may be configured to capture images of the surgical field (e.g., live feed) during a surgical procedure. For example, the visualization system may capture images of surgical instruments in the surgical field as the surgical instruments perform steps of a surgical procedure. Images captured by the visualization system 15008 can be displayed by any of the display devices disclosed herein, such as an augmented reality (AR) display device, to assist surgical personnel during surgical procedures.

[0119] In some embodiments, as described above, surgical instruments may be communicably connected to a surgical hub (for example, device / instrument 21 may be connected to the surgical hub 56 in Figure 5). Thus, the surgical hub 15002 may be configured to receive instrument data from the surgical instrument relating to various sensed parameters and operating settings of the instrument. Based on the instrument data received by the surgical hub 15002, the surgical hub 15002 can determine the operating parameters of the instrument. For example, based on instrument data received from various surgical instruments disclosed herein, the surgical hub 15002 can determine operating parameters such as speed, force, firing rate, firing force, startup state, power level, startup time, and energy mode.

[0120] In some embodiments, the surgical hub 15002 can be configured to identify interactions and potential interactions between surgical instruments and other objects based on data from the tracking system 15006 and / or instrument data received from the surgical instruments. Furthermore, potential interactions between surgical instruments and other objects may not be perceptible based solely on images captured by the imaging device 15010 of the visualization system 15008. Therefore, a user who relies on a display device (e.g., AR device 66) that displays only images captured from the imaging device 15010 may not be able to respond accurately to potential interactions. Accordingly, to assist the user, the surgical hub 15002 can be configured to cause the display device to display various notifications and other graphical indicators (e.g., overlays) related to interactions and / or potential interactions detected by the surgical hub.

[0121] In one embodiment, based on data from the tracking system 15006, the surgical hub 15002 can detect collisions or potential collisions of the tracked objects. For example, using any combination of the various tracking techniques disclosed herein (e.g., imaging device(s) 15010, structured optical sensor(s) 15012, LIDAR sensor(s) 15014, floor sensor(s) 15016, acoustic sensor(s) 15018, reference marker(s) 15020, user / device sensor(s) 15022, and GPS 15024), the surgical hub 15002 can detect potential collisions between a portion of a surgical instrument and a critical structure within the surgical field. In another example, the surgical hub 15002 can be configured to detect potential collisions between multiple surgical instruments. In yet another example, the surgical hub 15002 can be configured to detect potential collisions between various other objects within the surgical field. Detected potential collisions and / or detected collisions may not be within the field of view of the imaging device(s) 15010 and therefore may not be visible to the OR personnel. Based on detected potential collisions and / or detected collisions, the surgical hub 15002 may cause a display device (e.g., AR device 66) to display a notification, such as an overlay containing information about the collision. In one embodiment, the notification may include a warning and / or other instructions to avoid the collision. In another embodiment, the notification may include an overlay with a graphical representation of the objects involved in the collision. Thus, OR personnel can perceive and act upon potential collisions and collisions that are not within the field of view of the imaging device(s) 15010.

[0122] In another embodiment, based on data from the tracking system 15006, the surgical hub 15002 can detect unintended interactions of tracked objects. For example, similar to detecting potential collisions, the surgical hub 15002 can detect unintended interactions between a part of a surgical instrument and a critical structure in the surgical field. As another example, the surgical hub 15002 can detect unintended interactions between multiple surgical instruments. As yet another example, the surgical hub 15002 can detect unintended interactions between various other objects in the surgical field. Detected unintended interactions may not be within the field of view of the imaging device(s) 15010 and therefore may not be visible to the OR personnel. Based on the detected unintended interactions, the surgical hub 15002 can display notifications on a display device (e.g., AR device 66) such as an overlay of information about the unintended interaction, a warning and / or other command to avoid the interaction, and / or an overlay of a graphical representation of the objects involved in the interaction. Therefore, OR personnel can perceive and act upon unintended interactions that are not within the field of view of the imaging device(s) 15010. In some embodiments, the surgical hub 15002 can prevent instrument operation based on detected unintended interactions.

[0123] For example, a user may be using a unipolar energy device. The tracking system 15006 and / or the surgical hub 15002 may detect when the unipolar energy device is in close proximity to a metallic object (e.g., another surgical instrument, an object in the surgical field, etc.). Since activating the unipolar device in close proximity to a metallic object can cause an arc discharge, the surgical hub 15002 may determine that there is a potential unintended interaction between the unipolar device and the metallic object. Based on the detected unintended interaction, the surgical hub 15002 may cause the AR display device 66 to display an interaction overlay warning. In another embodiment, the surgical hub 15002 may prevent the unipolar energy device from activating. In yet another embodiment, the surgical hub 15002 may display an overlay instructing the user to redirect the energy direction towards the intended treatment zone.

[0124] In some embodiments, notifications, warnings, and / or overlays displayed based on detected potential collisions, detected collisions, detected unintended interactions, and other detected interactions between objects may include attributes of the objects involved in the interaction. The object attributes may be based on instrument data received by the surgical hub 15002 and / or on tracking data from the tracking system 15006. For example, the force, velocity, collision, and / or physical magnitude of the interaction between objects may be displayed as an overlay. In other embodiments, notifications, warnings, and / or overlays may include a graphic indicating the location of the interaction. A user viewing this graphic can then adjust the field of view of the imaging device to see the interaction.

[0125] In another embodiment, based on data from the tracking system 15006, the surgical hub 15002 may cause a display device (e.g., AR device 66) to display a graphic overlay to assist the user in performing steps of a surgical procedure. In one embodiment, the overlay may include a graphical representation and / or indicator of an object outside the field of view of the imaging device 15010 that captures an image of the surgical field. The graphic overlay may provide information about the location and / or other attributes of the object outside the field of view. For example, a surgeon may be performing a procedure using a circular stapler. The procedure may include attaching the device deck of the stapler to a separate anvil. The device deck of the stapler may be within the field of view, and the anvil may be outside the field of view (e.g., out of field of view based on camera angle, out of field of view because tissue is obstructing the anvil's view). The surgical hub 15002 may cause the display device to display a rendered image or other graphical representation of the anvil (e.g., indicating the off-image position of the anvil, overlaying a rendered image of the anvil on obstructing tissue). In another embodiment, the surgical hub 15002 may display a directional indicator overlay on the display device indicating the direction and / or path in which tissue may be manipulated in order to optimize the attachment of the anvil to the device deck. Thus, the overlay can help the surgeon perceive how objects outside the field of view of the imaging device 15010 may be manipulated in order to more easily achieve the desired outcome of the surgical procedure steps.

[0126] In another embodiment, based on data from the tracking system 15006, the surgical hub 15002 may display a graphic overlay by a display device (e.g., AR device 66) that can replace the need to use various instruments. For example, Figures 17A and 17B show an exemplary intraoperative display 15300 showing the surgical field 15302 during cutting of gastric tissue 15306, according to at least one non-limiting embodiment of the present disclosure. The intraoperative display 15000 may be displayed by any of the display devices (e.g., AR device 66) disclosed herein. Referring to Figure 17A, a bougie tube 15310 has been inserted into the patient and is providing a guide as the surgeon uses an endocutter 15312 assisted by a surgical gripper 15304 to make a cutting line 15308 in the gastric tissue 15306. Now referring to Figure 17B, the bougie tube 15310 is no longer in use. Alternatively, in one embodiment, a graphical overlay of a virtual bougie 15316 can be displayed on the intraoperative display 15300 to provide the surgeon with a guide for cutting the gastric tissue 15306. In another embodiment, a graphical overlay of a cutting line guide 15314 can be displayed on the intraoperative display 15300 to provide the surgeon with a guide. Thus, the graphical overlays 15314, 15316 can replace the need to physically place the bougie tube 15310 inside the patient.

[0127] Figure 18 illustrates Method 15400 for augmented reality visualization of surgical systems in some non-limiting aspects of the present disclosure. Method 15400 can be implemented by surgical systems, surgical hubs, tracking systems, visualization systems, augmented systems, AR devices, any of their components, and any other combination of any other devices and systems disclosed herein, such as surgical systems 1, 2, 50, 52, 15000, surgical hubs 6, 56, 5104, 15002, tracking system 15006, visualization systems 8, 15008, communication system 63, augmentation system 83, and AR devices 66, 84.

[0128] According to Method 15400, a first camera of a first visualization system can capture an image of an object in the surgical field (15402), where a first portion of the object is outside the field of view of the first camera. A tracking system can track the position of a second portion of the object (15404). A surgical hub can determine the attributes of the object based on the tracked position of the second portion of the object (15406), where the attributes of the object relate to the first portion of the object outside the camera's field of view. An augmented reality display device can display the captured image of the object in the surgical field and graphics based on the attributes of the object (15408). In one embodiment, the object may include a surgical instrument, patient tissue, a user, or a combination thereof.

[0129] According to one aspect of method 15400, determining the attributes of an object based on the tracked position of a second part of the object (15406) ​​may include determining the position of a first part of the object. Furthermore, displaying a graphic (15408) may include an augmented reality display device displaying a rendered image of the first part of the object.

[0130] In another aspect of Method 15400, the tracking system may include a visualization system. Furthermore, the visualization system may include a second camera. Tracking the location of a second part of an object (15404) may include the second camera capturing an image of the second part of the object. The second part of the object may be outside the field of view of the first camera. In another aspect of Method 15400, tracking the location of a second part of an object (15404) may include tracking the second part of the object using structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof.

[0131] In another aspect of Method 15400, the tracking system can track the location of a structure within the surgical field. Furthermore, determining the attributes of an object based on the tracked location of a second part of the object (15406) ​​may include the surgical hub identifying interactions between a first part of the object and the structure. In one aspect, an augmented reality display device may display graphics based on the location of the structure. In another aspect, displaying graphics based on the location of the structure may include displaying alerts based on identified interactions between a first part of the object and the structure. In yet another aspect, displaying graphics based on the location of the structure may include displaying interaction forces, interaction velocities, collisions between the first part of the object and the structure, the energization status of the object, time, or a combination thereof.

[0132] In another aspect of method 15400, the object may include a surgical instrument with a reference marker. In this aspect, tracking the position of a second part of the object (15404) may include a tracking system tracking the reference marker. In another aspect, a surgical hub may determine the range of motion of the surgical instrument based on the tracked reference marker. Furthermore, displaying graphics (15408) may include an augmented reality display device displaying a rendered image representing the range of motion of the surgical instrument.

[0133] Display of device-specific information and management of device usage via the network. As described throughout this disclosure, various devices and instruments can be used to perform surgical procedures. These devices can vary widely. For example, a device may have different device types and different device versions, each with different characteristics and applications. In some cases, the characteristics and applications of a device may be updated by the device manufacturer. Furthermore, the device manufacturer may develop new technologies for existing devices or release software updates related to device operation. In other cases, a device may be recalled by the manufacturer. In yet another case, there may be counterfeit devices or components of counterfeit devices that should not be used. Thus, there is a wealth of device identification-related information that OR personnel need to know when using devices for surgical procedures.

[0134] Furthermore, there is a wealth of device operation-related information that OR personnel must consider when using devices. For example, device performance may degrade over time based on repeated use. Another example is that devices may be overused or misused during surgical procedures. Moreover, devices may sense information that users may not be aware of or know how to easily access. Therefore, there is a need for devices, systems, and methods to manage device-related information and to ensure that users can easily access relevant device-related information.

[0135] In various embodiments, devices, systems, and methods for managing device-related information are disclosed herein. As described above, devices and surgical instruments may be communicatively connected to a surgical hub (for example, device / instrument 21 may be connected to the surgical hub 56 in Figure 5). Thus, the surgical hub may be configured to receive device-related information from various devices used with various surgical systems. Furthermore, the surgical hub may be communicatively coupled to a hospital network and / or a device manufacturer's network. For example, referring to Figure 5, a computer-implemented interactive surgical system 50 may include one or more surgical systems 52, each including at least one surgical hub 56 that communicates with a cloud 54 which may include a remote server 63. In one embodiment, the cloud 54 and / or the remote server 63 may be associated with a hospital network. The hospital network may communicate with a device manufacturer database. In another embodiment, the cloud 54 and / or the remote server 63 may be associated with a device manufacturer database.

[0136] In some embodiments, devices / instruments 21 connected to the surgical hub 56 can be authorized based on communication with the hospital network and / or the device manufacturer database. The hospital network can be configured to determine whether the connected device / instrument 21 is authorized. For example, a counterfeit device attempting to connect to the surgical hub 56 may not be authorized. In one embodiment, the hospital network can communicate with the manufacturer database to determine that the counterfeit device is not authorized. As another example, a recalled device attempting to connect to the surgical hub 56 may not be authorized. Unauthorized devices / instruments 21 may be prevented from being used, for example, by the surgical hub 56.

[0137] In one embodiment, the authorization of the device / device instrument 21 can be verified during the surgical procedure. In another embodiment, the authorization of the device / device can be verified when the device / device 21 and / or components of the device (e.g., reload cartridge, replacement components) are stocked. In yet another embodiment, the surgical hub 56 may be configured to allow the procedure to proceed even if the device / device 21 is not authorized. For example, if the lack of authorization is due to a hospital network outage, the procedure may be permitted to proceed without authorization of the device / device.

[0138] In some embodiments, the connected device / instrument 21 may store information relating to the techniques, applications, and / or software updates for using the device. This information may be communicated to the surgical hub 56 and stored by the hospital network (e.g., server 63). In other embodiments, information relating to the techniques, applications, and / or software updates for using the device may be accessed on the device manufacturer's database when the device is connected. Instructions and / or intended use for the device / instrument may be presented to the device user via a display device (e.g., AR device 66).

[0139] In some embodiments, the hospital network and / or device manufacturer database may store information regarding recommended and / or intended device use. This device use information can be used to determine whether a particular device / instrument 21 has exceeded its recommended use. For example, the device use information may include the maximum recommended use over a specific period (e.g., a device may not be intended to be used for longer than a specified period, or a device may not be intended to be activated more than a specified number of times over a specific period). As another example, the device use information may include the maximum recommended number of activations and / or maximum usage time over the course of the device's lifecycle. As yet another example, the device use information may include the intended use of a particular device / instrument 21. Based on this device use information, the surgical hub 56 may be configured to alert the user of detected overuse and / or misuse (e.g., via a display device such as an AR device 66). In other embodiments, the surgical hub 56 may be configured to prevent further use of a device based on the device use information stored by the hospital network and / or device manufacturer database.

[0140] In various embodiments, apparatuses, systems, and methods for enabling a user to easily access relevant device-related information are disclosed herein. Referring still to Figure 5, and also to Figure 8, a user using a device / instrument 21 connected to a surgical hub 56 can request a display device (e.g., hub display 65, instrument display 50, AR device 66, etc.) to display device information related to the device / instrument 21. For example, a surgeon can request that device information be displayed by providing an oral prompt detected by a microphone associated with the surgical hub 56 (e.g., the surgeon could say, "Show me the information"). The user's request to display device information can cause the surgical hub 56 to cause the display device to display information related to the device / instrument 21.

[0141] In various embodiments, the information displayed in relation to the device / instrument 21 may include information related to the device's usage history and other information related to the device's current operation. In some embodiments, the displayed information may vary depending on the type of device. In one embodiment, if the device / instrument 21 is an energy device, the displayed information may include, for example, the number of firings, the total firing time, the residual temperature in the jaws, the estimated fit of the device, device-specific calibration and / or characterization information that may affect best use, the device parameters in question, or a combination thereof. In another embodiment, if the device / instrument 21 is an endocutter, the displayed information may include, for example, the number of firings, estimated anvil curvature correction information, high / low interstitial gaps based on firings on the test skin being constructed, the device parameters in question (e.g., maximum joint angle, jaw temperature, etc.), or a combination thereof. Thus, the user of the device may be able to easily determine (e.g., based on the displayed usage history) whether the device is approaching or has exceeded its recommended lifespan. The user may also be able to easily access important parameters related to the device's operation to assist in decision-making during surgical procedures.

[0142] Various additional aspects of the subject matter described herein are illustrated in the following numbered examples. Example 1: A method for augmented reality visualization of a surgical system, comprising: capturing an image of an object in the surgical field with a first camera of a first visualization system, wherein a first portion of the object is outside the field of view of the first camera; tracking the position of a second portion of the object with a tracking system; determining the attributes of the object based on the tracked position of the second portion of the object with a surgical hub, wherein the attributes of the object relate to the first portion of the object outside the field of view of the camera; and displaying the captured image of the object in the surgical field and graphics based on the attributes of the object with an augmented reality display device, wherein the object includes a surgical instrument, patient tissue, or a user, or a combination thereof.

[0143] Example 2: The method according to Example 1, wherein determining the attributes of an object based on the tracked position of a second part of the object includes determining the position of a first part of the object, and displaying graphics includes displaying a rendered image of the first part of the object by an augmented reality display device.

[0144] Example 3: The method according to any one of Examples 1 to 2, wherein the tracking system comprises a visualization system, the visualization system comprises a second camera, and tracking the position of a second part of an object includes capturing an image of the second part of the object by the second camera, the second part of the object being outside the field of view of the first camera.

[0145] Example 4: The method according to any one of Examples 1 to 3, wherein tracking the position of a second part of an object by a tracking system includes tracking the second part of an object using structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof.

[0146] Example 5: The method according to any one of Examples 1 to 4, further comprising tracking the location of a structure within the surgical field by a tracking system, determining the attributes of the object based on the tracked location of a second part of the object, and identifying the interaction between a first part of the object and the structure by a surgical hub.

[0147] Example 6: The method according to any one of Examples 1 to 5, further comprising displaying graphics based on the position of a structure using an augmented reality display device.

[0148] Example 7: The method according to any one of Examples 1 to 6, wherein displaying graphics based on object attributes includes displaying alerts based on identified interactions by an augmented reality display device.

[0149] Example 8: The method according to any one of Examples 1 to 7, wherein displaying graphics based on object attributes includes displaying interaction forces, interaction velocities, collisions of first parts and structures of an object, energized state of an object, time, or a combination thereof by an augmented reality display device.

[0150] Example 9: The method according to any one of Examples 1 to 8, wherein the object includes a surgical instrument with a reference marker, and tracking the position of a second part of the object includes tracking the reference marker by a tracking system.

[0151] Example 10: The method according to any one of Examples 1 to 9, further comprising determining the range of motion of a surgical instrument based on tracked reference markers by a surgical hub, and displaying graphics by an augmented reality display device, the method comprising displaying graphics representing the range of motion of the surgical instrument.

[0152] Example 11: A surgical system for mixed reality visualization, comprising: a first visualization system comprising a first camera configured to capture an image of an object in a surgical field, wherein the first portion of the object is outside the camera's field of view; a first tracking system configured to track the position of a second portion of the object; a surgical hub configured to determine the attributes of an object based on the tracked position of the second portion of the object, wherein the attributes of the object are related to the first portion of the object outside the camera's field of view; and an augmented reality display device configured to display a captured image of an object in a surgical field and graphics based on the attributes of the object, wherein the object includes a surgical instrument, patient tissue, a user, or a combination thereof.

[0153] Example 12: The system according to Example 11, wherein the object attributes include the position of a first part of the object, and the graphics include a rendered image of the first part of the object.

[0154] Example 13: The system according to any one of Examples 11 to 12, wherein the tracking system comprises a visualization system, the visualization system comprising a second camera configured to capture an image of a second portion of an object, the second portion of the object being outside the field of view of the first camera of the first visualization system.

[0155] Example 14: The tracking system is one of the systems from Examples 11-13, using structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof.

[0156] Example 15: The tracking system is configured to track the location of a structure within the surgical field, the surgical hub is configured to identify interactions between a first part of an object and a structure, and the object's attributes include the identified interactions, as described in any one of Examples 11-14.

[0157] Example 16: The augmented reality display device is configured to display graphics based on the position of a structure, as described in any one of Examples 11-15.

[0158] Example 17: A system described in any one of Examples 11-16, in which graphics based on object attributes include alerts based on identified interactions.

[0159] Example 18: A system according to any one of Examples 11-17, wherein the graphics based on object attributes include interaction forces, interaction velocities, a display of collisions between first parts and structures of the object, the energized state of the object, time, or a combination thereof.

[0160] Example 19: The system according to any one of Examples 11-18, wherein the object includes another surgical instrument, the surgical instrument is equipped with a reference marker, and the tracking system is configured to track the reference marker.

[0161] Example 20: The surgical hub is configured to determine the range of motion of a surgical instrument based on tracked reference markers, the object attributes include the range of motion of the surgical instrument, and the graphics based on the object attributes include graphics representing the range of motion of the surgical instrument, as described in any one of Examples 11-19.

[0162] While several forms have been shown and described, it is not the applicant's intention to limit or restrict the attached claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be implemented and will be conceived by those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element related to the described form can be alternatively described as a means for providing the function performed by that element. Also, while materials are disclosed with respect to specific components, other materials may be used. Therefore, it should be understood that the above description and the attached claims are intended to cover all such modifications, combinations, and variations as being included within the scope of the disclosed forms. The attached claims are intended to cover all such modifications, variations, alterations, substitutions, alterations, and equivalents.

[0163] The detailed descriptions above have described various forms of apparatus and / or processes using block diagrams, flowcharts and / or embodiments. To the extent that such block diagrams, flowcharts and / or embodiments include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flowcharts and / or embodiments can be individually and / or collectively implemented by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some or all of the forms disclosed herein can be equivalently implemented on integrated circuits as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code falls within the scope of the skills of those skilled in the art in light of this disclosure. Furthermore, as will be understood by those skilled in the art, the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and the specific forms of the subject matter described herein are applicable regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0164] Instructions used to program logic to implement various disclosed embodiments may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, instructions may be distributed over a network or by other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but are not limited to floppy diskettes, optical disks, compact disks, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage used for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, non-temporary computer-readable media may include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0165] When used in any aspect of this specification, the term “control circuit” can mean, for example, hardwired circuits, programmable circuits (e.g., computer processors, processing units, processors, microcontrollers, microcontroller units, controllers, digital signal processors (DSPs), programmable logic devices (PLDs), programmable logic arrays (PLAs), or field-programmable gate arrays (FPGAs) including one or more individual instruction processing cores), state-machine circuits, firmware that stores instructions executed by programmable circuits, and any combination thereof. Control circuits can be embodied collectively or individually as circuits that form part of a larger system, such as an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, or a smartphone. Accordingly, as used herein, “control circuit” includes, but is not limited to, an electrical circuit having at least one separate electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application-specific integrated circuit, an electrical circuit forming a general-purpose computing device configured by a computer program (e.g., a general-purpose computer configured by a computer program that performs at least partially the processes and / or devices described herein, or a microprocessor configured by a computer program that performs at least partially the processes and / or devices described herein), an electrical circuit forming a memory device (e.g., in the form of random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electric installation). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog form, digital form, or some combination thereof.

[0166] When used in any aspect of this specification, the term “logic” may mean an application, software, firmware, and / or circuit configured to perform any of the operations described above. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-temporary computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and / or hardcoded (e.g., non-volatile) data in a memory device.

[0167] When used in any aspect of this specification, the terms “component,” “system,” “module,” etc., may refer to a control circuit, a computer-related entity, hardware, a combination of hardware and software, software, or running software.

[0168] Where used in any aspect of this specification, “algorithm” means a self-consistent sequence of steps leading to a desired result, and “step” means the manipulation of physical quantities and / or logical states that can take the form of electrical or magnetic signals, which are not necessarily required but can be stored, transferred, combined, compared, and otherwise manipulated. These signals are commonly referred to as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities, or simply are convenient labels applied to these quantities and / or states.

[0169] A packet-switched network is one example of a network. Communication devices can communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol is the Ethernet communication protocol, which can enable communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of the Ethernet standard. Alternatively or additionally, communication devices can communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices can communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol conforms to or may be compatible with standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol conforms to or may be compatible with the ATM standard and / or later versions of this standard, published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0". Naturally, different and / or later developed connection-oriented network communication protocols are equally construed herein.

[0170] Unless otherwise explicitly stated, as is evident from the foregoing disclosures, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the foregoing disclosures should be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage, transmission, or display device.

[0171] One or more components may be referred to herein as “configured to,” “configurable to,” “operable / operative to,” “adapted / adaptable,” “able to,” “conformable / conformed to,” and so on. Those skilled in the art will understand that “configured to” generally encompasses active components and / or inactive components and / or standby components, unless the context should interpret it otherwise.

[0172] The terms “proximal” and “distal” are used herein in reference to the clinician operating the handle portion of a surgical instrument. “Proximal” refers to the part closest to the clinician, and “distal” refers to the part further away from the clinician. For convenience and clarity, spatial terms such as “vertical,” “horizontal,” “up,” and “down” may be used herein in reference to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0173] Those skilled in the art will generally understand that the terms used herein, and especially in the appended claims (e.g., the text of the appended claims), are generally intended to be "open" terms (for example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "includes" should be interpreted as "includes but is not limited to"). Furthermore, those skilled in the art will understand that if a particular number is intended in an introduced claim recitation, such intent is clearly stated in the claim, and if such statement is not present, such intent does not exist. For example, to aid understanding, subsequent appended claims may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be interpreted as suggesting that any particular claim containing such introduced claim description is limited to claims containing only one such description, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" (for example, "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using a definite article.

[0174] In addition, even if a specific number is explicitly stated in the introduced claim, it will be recognized by those skilled in the art that such a statement should typically be interpreted as meaning at least the number stated (for example, if there is a statement that is simply “two descriptions” without any other modifiers, it generally means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to “at least one of A, B, and C, etc.” is used, such a notation is generally intended to be understood in a way that those skilled in the art will understand (for example, “a system having at least one of A, B, and C” is not limited to systems having only A, only B, only C, both A and B, both A and C, both B and C and / or all of A, B and C, etc.). When expressions similar to "at least one of A, B, or C" are used, such expressions are generally intended to be understood in a way that a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Furthermore, a person skilled in the art will understand that, typically, any disjunctive word and / or phrase representing two or more selective terms should be understood, whether in the specification, claims, or drawings, as intended to include the possibility of including one of those terms, any of those terms, or both of those terms, unless the context requires a different interpretation. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0175] With respect to the attached claims, those skilled in the art will understand that the operations cited herein may generally be performed in any order. Furthermore, while various operations are shown in sequence(s), it should be understood that the operations may be performed in any order other than those shown, or simultaneously. Examples of such alternative orderings may include repetition, alternation, interruption, reordering, augmentation, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context should imply otherwise. Moreover, terms such as “responsive to,” “related to,” or other past tense adjectives are generally not intended to exclude such variations, unless the context should imply otherwise.

[0176] It is worth noting that any reference to “one aspect,” “aspect,” “example,” or “example” means that the specific feature, structure, or characteristic described in relation to that aspect is included in at least one aspect. Therefore, the phrases “in one aspect,” “in aspect,” “example,” and “example” found in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, specific features, structures, or characteristics can be combined in any preferred manner in one or more aspects.

[0177] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material does not conflict with this Specified. Disclosures expressly stated herein, both in themselves and to the extent required, shall supersede any conflicting statements incorporated herein by reference. Any material, or any part thereof, that is referred to as being incorporated herein by reference but conflicts with current definitions, views, or other disclosures contained herein shall be incorporated only to the extent that there is no conflict between the incorporated material and the current disclosures.

[0178] In summary, the numerous benefits that can be obtained as a result of using the concepts described herein have been described. The above descriptions of one or more forms are presented for illustrative and explanatory purposes only. They are not intended to be comprehensive or to be limited to the exact forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms have been selected and described to illustrate the principle and practical applications, thereby enabling a person skilled in the art to utilize the various forms, along with various modifications, for specific conceivable uses. The claims presented herein are intended to define the overall scope.

[0179] [Implementation Method] (1) A method for augmented reality visualization of surgical systems, The first visualization system captures an image of an object within the surgical field using a first camera, wherein a first portion of the object is outside the field of view of the first camera. The tracking system tracks the position of the second part of the object, The surgical hub determines the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the camera's field of view. The augmented reality display device includes displaying the captured image of the object in the surgical field and graphics based on the attributes of the object, The object includes a surgical instrument, patient tissue, or user, or a combination thereof, or a method. (2) Determining the attributes of the object based on the tracked position of the second part of the object includes determining the position of the first part of the object, The method according to Embodiment 1, wherein displaying the graphic includes displaying a rendered image of the first portion of the object by the augmented reality display device. (3) The tracking system comprises the visualization system, The visualization system includes a second camera, Tracking the position of the second portion of the object includes capturing an image of the second portion of the object with the second camera. The method according to Embodiment 1, wherein the second portion of the object is located outside the field of view of the first camera. (4) The method according to Embodiment 1, wherein tracking the position of the second part of the object by the tracking system includes tracking the second part of the object using structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof. (5) The tracking system further includes tracking the location of structures within the surgical field, The method according to Embodiment 1, wherein determining the attributes of the object based on the tracked position of the second portion of the object includes identifying the interaction between the first portion of the object and the structure by the surgical hub.

[0180] (6) The method according to Embodiment 5, further comprising displaying graphics based on the position of the structure using the augmented reality display device. (7) The method of Embodiment 5, wherein displaying the graphics based on the attributes of the object includes displaying an alert based on the identified interaction by the augmented reality display device. (8) The method of Embodiment 5, wherein displaying the graphics based on the attributes of the object includes displaying the force of the interaction, the velocity of the interaction, a collision of the first part and the structure of the object, the energized state of the object, time, or a combination thereof, by the augmented reality display device. (9) The object comprises a surgical instrument including a reference marker, The method according to Embodiment 1, wherein tracking the position of the second portion of the object includes tracking the reference marker by the tracking system. (10) The surgical hub further includes determining the range of motion of the surgical instrument based on the tracked reference marker, The method according to embodiment 9, wherein displaying the graphic includes displaying a graphic representing the range of motion of the surgical instrument using the augmented reality display device.

[0181] (11) A surgical system for augmented reality visualization, A first visualization system comprising a first camera configured to capture an image of an object within the surgical field, wherein a first portion of the object is outside the field of view of the camera, A first tracking system configured to track the position of a second part of the object, A surgical hub configured to determine the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the camera, The system comprises an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object is a surgical system including surgical instruments, patient tissue, a user, or a combination thereof. (12) The attributes of the object include the position of the first part of the object, The surgical system according to embodiment 11, wherein the graphic includes a rendered image of the first part of the object. (13) The tracking system comprises the visualization system, The visualization system includes a second camera configured to capture an image of the second portion of the object, The surgical system according to Embodiment 11, wherein the second portion of the object is outside the field of view of the first camera of the first visualization system. (14) The surgical system according to Embodiment 11, wherein the tracking system uses structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof. (15) The tracking system is configured to track the location of structures within the surgical field, The surgical hub is configured to identify the interaction between the first part of the object and the structure, The surgical system according to Embodiment 11, wherein the attributes of the object include the identified interactions.

[0182] (16) The surgical system according to Embodiment 15, wherein the augmented reality display device is configured to display graphics based on the position of the structure. (17) The surgical system according to Embodiment 15, wherein the graphic based on the attributes of the object includes an alert based on the identified interaction. (18) The surgical system according to Embodiment 15, wherein the graphic based on the attributes of the object includes the force of the interaction, the velocity of the interaction, a representation of the collision of the first part of the object and the structure, the energized state of the object, time, or a combination thereof. (19) The object includes the surgical instrument, The aforementioned surgical instrument includes a reference marker, The surgical system according to embodiment 11, wherein the tracking system is configured to track the reference marker. (20) The surgical hub is configured to determine the range of motion of the surgical instrument based on the tracked reference marker, The attributes of the object include the range of motion of the surgical instrument, The surgical system according to embodiment 19, wherein the graphic based on the attributes of the object includes a graphic representing the range of motion of the surgical instrument.

Claims

1. A method for augmented reality visualization of surgical systems, The first visualization system captures an image of an object within the surgical field using a first camera, wherein a first portion of the object is outside the field of view of the first camera. The tracking system tracks the position of the second part of the object, The surgical hub determines the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera. The augmented reality display device includes displaying the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, or a user, or a combination thereof. The tracking system further includes tracking the position of structures within the surgical field, A method for determining the attributes of an object based on the tracked position of the second portion of the object, comprising identifying the interaction between the first portion of the object and the structure by the surgical hub.

2. Determining the attributes of the object based on the tracked position of the second portion of the object includes determining the position of the first portion of the object. The method according to claim 1, wherein displaying the graphic includes displaying a rendered image of the first portion of the object by the augmented reality display device.

3. A method for augmented reality visualization of a surgical system, The first visualization system captures an image of an object within the surgical field using a first camera, wherein a first portion of the object is outside the field of view of the first camera. The tracking system tracks the position of the second part of the object, The surgical hub determines the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera. The augmented reality display device includes displaying the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, or a user, or a combination thereof. The tracking system includes the visualization system, The visualization system includes a second camera, Tracking the position of the second portion of the object includes capturing an image of the second portion of the object with the second camera, The method wherein the second portion of the object is outside the field of view of the first camera.

4. A method for augmented reality visualization of a surgical system, The first visualization system captures an image of an object within the surgical field using a first camera, wherein a first portion of the object is outside the field of view of the first camera. The tracking system tracks the position of the second part of the object, The surgical hub determines the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera. The augmented reality display device includes displaying the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, or a user, or a combination thereof. A method for tracking the position of the second portion of the object by the tracking system, comprising tracking the second portion of the object using structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof.

5. The method according to claim 1, further comprising displaying graphics based on the position of the structure using the augmented reality display device.

6. The method according to claim 1, wherein displaying the graphics based on the attributes of the object includes displaying an alert based on the identified interaction by the augmented reality display device.

7. The method according to claim 1, wherein displaying the graphics based on the attributes of the object includes displaying, by the augmented reality display device, the force of the interaction, the velocity of the interaction, a display of collisions between the first part and the structure of the object, the energized state of the object, time, or a combination thereof.

8. A method for augmented reality visualization of a surgical system, The first visualization system captures an image of an object within the surgical field using a first camera, wherein a first portion of the object is outside the field of view of the first camera. The tracking system tracks the position of the second part of the object, The surgical hub determines the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera. The augmented reality display device includes displaying the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, or a user, or a combination thereof. The object comprises a surgical instrument including a reference marker, A method for tracking the position of the second portion of the object, comprising tracking the reference marker by the tracking system.

9. The surgical hub further includes determining the range of motion of the surgical instrument based on the tracked reference marker, The method according to claim 8, wherein displaying the graphic includes displaying a graphic representing the range of motion of the surgical instrument using the augmented reality display device.

10. A surgical system for augmented reality visualization, A first visualization system comprising a first camera configured to capture an image of an object within the surgical field, wherein a first portion of the object is outside the field of view of the first camera, A first tracking system configured to track the position of a second part of the object, A surgical hub configured to determine the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera, The system comprises an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, users, or a combination thereof. The tracking system is configured to track the position of structures within the surgical field. The surgical hub is configured to identify the interaction between the first part of the object and the structure, The attributes of the object include the identified interactions in the surgical system.

11. The attributes of the object include the position of the first part of the object, The surgical system according to claim 10, wherein the graphic includes a rendered image of the first portion of the object.

12. A surgical system for augmented reality visualization, A first visualization system comprising a first camera configured to capture an image of an object within the surgical field, wherein a first portion of the object is outside the field of view of the first camera, A first tracking system configured to track the position of a second part of the object, A surgical hub configured to determine the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera, The system comprises an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, users, or a combination thereof. The tracking system includes the visualization system, The visualization system includes a second camera configured to capture an image of the second portion of the object, A surgical system in which the second portion of the object is located outside the field of view of the first camera of the first visualization system.

13. A surgical system for augmented reality visualization, A first visualization system comprising a first camera configured to capture an image of an object within the surgical field, wherein a first portion of the object is outside the field of view of the first camera, A first tracking system configured to track the position of a second part of the object, A surgical hub configured to determine the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera, The system comprises an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, users, or a combination thereof. The tracking system is a surgical system that uses structured light sensors, light detection and ranging (LIDAR) sensors, radio frequency identification (RFID), global positioning system (GPS) tracking, voice beacons, non-visual light tracking, or a combination thereof.

14. The surgical system according to claim 10, wherein the augmented reality display device is configured to display graphics based on the position of the structure.

15. The surgical system according to claim 10, wherein the graphic based on the attributes of the object includes an alert based on the identified interaction.

16. The surgical system according to claim 10, wherein the graphic based on the attributes of the object includes the force of the interaction, the velocity of the interaction, a representation of the collision of the first part of the object and the structure, the energized state of the object, time, or a combination thereof.

17. A surgical system for augmented reality visualization, A first visualization system comprising a first camera configured to capture an image of an object within the surgical field, wherein a first portion of the object is outside the field of view of the first camera, A first tracking system configured to track the position of a second part of the object, A surgical hub configured to determine the attributes of the object based on the tracked position of the second portion of the object, wherein the attributes of the object relate to the first portion of the object outside the field of view of the first camera, The system comprises an augmented reality display device configured to display the captured image of the object in the surgical field and graphics based on the attributes of the object, The aforementioned object includes surgical instruments, patient tissue, users, or a combination thereof. The object includes the surgical instrument, The aforementioned surgical instrument includes a reference marker, The tracking system is a surgical system configured to track the reference marker.

18. The surgical hub is configured to determine the range of motion of the surgical instrument based on the tracked reference marker, The attributes of the object include the range of motion of the surgical instrument, The surgical system according to claim 17, wherein the graphic based on the attributes of the object includes a graphic representing the range of motion of the surgical instrument.

Images