Risk-based prioritization of display methods in the surgical field

Abstract

The surgical system is for use in a surgical procedure. The surgical system includes a surgical instrument configured to treat tissue in the surgical procedure, an imaging device, a display configured to show a live stream of a surgical field of the surgical procedure, the live stream being captured by the imaging device, and a control module. The control module is configured to detect a surgical risk, assign a display priority to the surgical risk, and determine a display placement of the surgical risk based on the display priority, the display placement including overlaying an alert feature on the live stream.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 174,674, titled "HEADS UP DISPLAY," filed on April 14, 2021, and U.S. Provisional Patent Application No. 63 / 284,326, titled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS," filed on November 30, 2021, under 35 U.S.C. § 119(e). The entire disclosure of each of these applications is incorporated herein by reference in its entirety.

Background Art

[0002] The present disclosure relates to devices, systems, and methods for providing an augmented reality interactive experience during a surgical procedure. During a surgical procedure, it would be desirable to provide an augmented reality interactive experience of the real - world environment in which objects existing in the real world are enhanced by overlaying computer - generated perceptual information across multiple sensory modalities, including, at times, vision, hearing, touch, proprioception, and smell. In the context of the present disclosure, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer - generated visual, auditory, tactile, proprioceptive, olfactory, or other sensory information onto the real - world images of the surgical field and the instruments or other objects appearing in the surgical field. The images may be streamed in real - time or may be still images.

[0003] Real-world surgical instruments include a variety of surgical devices, including energy, staplers, or combinations of energy and staplers. Energy-based medical devices include, but are not limited to, radio frequency (RF) based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices are surgical instruments used to cut and staple tissue in a variety of surgical procedures, including obesity, thoracic, colorectal, obstetric and gynecological, urological, and general surgery. [Overview of the project] [Means for solving the problem]

[0004] In various examples, surgical systems are used in surgical procedures. A surgical system comprises surgical instruments configured to treat tissue in a surgical procedure, an imaging device, a display configured to show a live stream of the surgical field of the surgical procedure, the display of which the live stream is captured by the imaging device, and a control module. The control module is configured to detect surgical risks, assign display priorities to surgical risks, and determine the display placement of surgical risks based on the display priorities, wherein the display placement includes overlaying alert features on the live stream.

[0005] In various examples, surgical systems are used in surgical procedures. A surgical system comprises surgical instruments configured to treat tissue in a surgical procedure, an imaging device, a display configured to show a live stream of the surgical field of the surgical procedure, the display of which the live stream is captured by the imaging device, and a control module. The control module is configured to detect surgical risks, assign severity levels to surgical risks, and determine a display arrangement of surgical risks based on the severity level of the surgical risks, wherein the display arrangement includes alert features. [Brief explanation of the drawing]

[0006] The various embodiments described herein with respect to both configuration and operation methods, along with their further purposes and advantages, can be best understood by referring to the following description in conjunction with the accompanying drawings. [Figure 1] This is a block diagram of a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 2] This is a diagram of a surgical system used to perform surgical procedures in an operating room, according to one aspect of the present disclosure. [Figure 3] One aspect of the present disclosure is a visualization system, a robotic system, and a surgical hub paired with an intelligent instrument. [Figure 4] This figure shows a surgical data network, according to one aspect of the present disclosure, which includes a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This figure shows a computer-implemented interactive surgical system according to one aspect of the present disclosure. [Figure 6] This figure shows a surgical hub, including a plurality of modules connected to a modular control tower, according to one aspect of the present disclosure. [Figure 7] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 8] This figure shows an augmented reality (AR) system, according to one aspect of the present disclosure, which includes an intermediate signal coupler located in the communication path between an imaging module and a surgical hub display. [Figure 9] This figure shows an augmented reality (AR) device worn by a surgeon to communicate data to a surgical hub, according to one aspect of the present disclosure. [Figure 10]This figure shows a system for augmenting surgical instrument information using an augmented reality display, according to one aspect of the present disclosure. [Figure 11] This figure shows a timeline of a situational awareness surgical procedure according to one aspect of the present disclosure. [Figure 12] This figure shows a surgical visualization system according to at least one aspect of the present disclosure. [Figure 13] This flowchart shows the operation of an exemplary method for determining the display arrangement of competing surgical data seeking presentation on a display, according to at least one aspect of the present disclosure. [Figure 14] A flowchart shows the operation of an exemplary method 6020 for determining the display priority value of detected (6011) surgical data according to the method in Figure 13. [Figure 15] This is a flowchart illustrating the operation of a method for determining display priority values ​​for tissue tension and / or pressure parameters within a surgical anastomosis, according to at least one aspect of the present disclosure. [Figure 16] This is a flowchart illustrating the operation of a method for determining a display priority value based on a trigger event, according to at least one aspect of the present disclosure. [Figure 17] This is a flowchart illustrating the operation of the method according to at least one aspect of this disclosure. [Figure 18] This flowchart shows the operation of a method for automatic switching between live streams of the surgical field in a surgical procedure, according to at least one aspect of the present disclosure. [Figure 19] This is a flowchart illustrating the operation of a method for balancing system resources during a surgical procedure, according to at least one aspect of the present disclosure. [Figure 19A] This flowchart shows the operation of a method for transitioning between a static display mode and an active display mode based on surgical data, according to at least one aspect of the present disclosure. [Figure 19B]A flowchart showing the operation of a method for transitioning a visual representation of surgical data between a static display mode and an active display mode, according to at least one aspect of the present disclosure. [Figure 20] A flowchart showing the operation of a method for resolving display conflicts in a display arrangement, according to at least one aspect of the present disclosure. [Figure 21] A flowchart showing the operation of a method for dealing with tissue changes in a surgical procedure employing a surgical instrument, according to at least one aspect of the present disclosure. [Figure 22A] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 22B] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 22C] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 23A] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 23B] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 24] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 25] A diagram showing a display arrangement, according to at least one aspect of the present disclosure. [Figure 26] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 27A] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 27B] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 27C] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 28] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 29] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 30] A diagram showing a display arrangement according to the method of the present disclosure. [Figure 31]This is a flowchart illustrating the operation of a method for risk-based operation of display placement during surgical procedures, according to at least one aspect of the present disclosure. [Figure 32] This figure shows a display arrangement according to at least one aspect of the present disclosure. [Figure 33] This figure shows a display arrangement according to at least one aspect of the present disclosure. [Figure 34] This figure shows a display arrangement according to at least one aspect of the present disclosure.

[0007] Throughout the drawings, corresponding reference numerals indicate corresponding parts. The examples described herein illustrate various disclosed embodiments in one form, and such examples should not be construed as limiting the scope. [Modes for carrying out the invention]

[0008] The applicant of this application owns the following concurrently filed U.S. patent applications, the entirety of which is incorporated herein by reference: • U.S. Patent Application titled "METHOD FOR INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Agent Reference Number END9352USNP1 / 210120-1M U.S. Patent Application titled "UTILIZATION OF SURGICAL DATA VALUES AND SITUATIONAL AWARENESS TO CONTROL THE OVERLAY IN SURGICAL FIELD VIEW"; Agent Reference Number END9352USNP2 / 210120-2 • U.S. Patent Application entitled "Selective and adjustable mixed reality overlay insurgical field view"; Agent reference number END9352USNP3 / 210120-3 • U.S. Patent Application entitled "SYSTEMS AND METHODS FOR CONTROLLING SURGICAL DATA OVERLAY"; Agent Reference Number END9352USNP5 / 210120-5 U.S. Patent Application entitled "SYSTEMS AND METHODS FOR CHANGING DISPLAY OVERLAY OF SURGICAL FIELD VIEW BASED ON TRIGGERING EVENTS"; Agent Reference Number END9352USNP6 / 210120-6 • U.S. Patent Application entitled "CUSTOMIZATION OF OVERLAID DATA AND CONFIGURATION"; Agent Reference Number END9352USNP7 / 210120-7 U.S. Patent Application entitled "INDICATION OF THE COUPLE PAIR OF REMOTE CONTROLS WITH REMOTE DEVICES FUNCTIONS"; Agent Reference Number END9352USNP8 / 210120-8 U.S. Patent Application entitled "Cooperative Overlays of Interacting Instruments Which Resurface in Both Overlays Being Effected"; Agent Reference Number END9352USNP9 / 210120-9 U.S. Patent Application entitled "ANTICIPATION OF INTERACTIVE UTILIZATION OF COMMON DATA OVERLAYS BY DIFFERENT USERS"; Agent Reference Number END9352USNP10 / 210120-10 U.S. Patent Application entitled "MIXING DIRECTLY VISUALIZED WITH RENDERED ELEMENTS TO DISPLAY BLENDED ELEMENTS AND ACTIONS HAPPENING ON-SCREEN AND OFF-SCREEN"; Agent Reference Number END9352USNP11 / 210120-11 U.S. Patent Application entitled "SYSTEM AND METHOD FOR TRACKING A PORTION OF THE USER AS A PROXY FOR NON-MONITORED INSTRUMENT"; Agent Reference Number END9352USNP12 / 210120-12 U.S. Patent Application entitled "UTILIZING CONTEXTUAL PARAMETERS OF ONE OR MORE SURGICAL DEVICES TO PREDICT A FREQUENCY INTERVAL FOR DISPLAYING SURGICAL INFORMATION"; Agent Reference Number END9352USNP13 / 210120-13 U.S. Patent Application entitled "Cooperation Among Multiple Display Systems to Provide a Healthcare User Customized Information"; Agent Reference Number END9352USNP14 / 210120-14 • U.S. Patent Application entitled "INTRAOPERATIVE DISPLAY FOR SURGICAL SYSTEMS"; Agent reference number END9352USNP15 / 210120-15 • U.S. Patent Application entitled "ADAPTATION AND ADJUSTABILITY OR OVERLAID INSTRUMENT INFORMATION FOR SURGICAL SYSTEMS"; Agent Reference Number END9352USNP16 / 210120-16, and, U.S. Patent Application entitled "MIXED REALITY FEEDBACK SYSTEMS THAT COOPERATE TO INCREASE EFFICIENT PERCEPTION OF COMPLEX DATA FEEDS"; Agent reference number END9352USNP17 / 210120-17.

[0009] The applicant of this application owns the following U.S. patent applications, the entirety of which is incorporated herein by reference: U.S. Patent Application No. 16 / 209,423 (currently U.S. Patent Publication No. 2019 / 0200981(A1)), entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS" U.S. Patent Application No. 16 / 209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES" (currently published as U.S. Patent Application Publication No. 2019 / 0201046(A1)).

[0010] Before describing in detail the various embodiments of surgical devices and generators, it should be noted that the illustrative embodiments are not limited in their application or use to the details of the structure and arrangement of the components illustrated in the accompanying drawings and descriptions. The illustrative embodiments may be implemented or incorporated into other embodiments, variations, and modifications, and may be carried out or performed in various ways. Furthermore, unless otherwise specified, the terms and expressions used herein have been selected for the purpose of illustrating the illustrative embodiments for the convenience of the reader and are not intended to limit them. Furthermore, it should be understood that one or more embodiments, expressions of embodiments, and / or embodiments described below may be combined with any one or more other embodiments, expressions of embodiments, and / or embodiments described below.

[0011] Various embodiments apply to on-screen displays for surgical systems for various energy and surgical stapler-based medical devices. Energy-based medical devices include, but are not limited to, radio frequency (RF)-based unipolar and bipolar electrosurgical instruments, ultrasonic surgical instruments, combinations of RF electrosurgical instruments and ultrasonic instruments, and combinations of RF electrosurgical staplers and mechanical staplers. Surgical stapler devices include surgical staplers combined with electrosurgical devices and / or ultrasonic devices. Embodiments of ultrasonic surgical devices may be configured, for example, to transversely incise and / or coagulate tissue during surgical procedures. Embodiments of electrosurgical devices may be configured, for example, to transversely incise, coagulate, seal, weld and / or dry tissue during surgical procedures. Embodiments of surgical stapler devices may be configured to transversely incise and staple tissue during surgical procedures, and in some embodiments, surgical stapler devices may be configured to deliver RF energy to tissue during surgical procedures. Electrosurgical devices are configured to deliver therapeutic and / or non-therapeutic RF energy to tissue. Elements of surgical staplers, electrosurgical devices, and ultrasound devices can be used in combination within a single surgical instrument.

[0012] In various embodiments, the Disclosure provides the OR team with on-screen displays of real-time information during surgical procedures. According to various embodiments of the Disclosure, many novel and unique on-screen displays are provided for displaying various visual information feedback to the OR team on screen. According to the Disclosure, the visual information may include one or more of various visual media, with or without sound. Generally, the visual information includes still photographs, moving photographs, video or audio recordings, graphic art, visual aids, models, displays, visual representation services, and support processes. The visual information may be communicated on any number of display options, such as, for example, a primary OR screen, the energy or surgical stapler device itself, a tablet, augmented reality glasses, etc.

[0013] In various embodiments, this disclosure provides a list of many potential options for communicating visual information to an OR team in real time without overwhelming the OR team with too much visual information. For example, in various embodiments, this disclosure provides on-screen displays of visual information that enable a surgeon, or other member of the OR team, to selectively activate on-screen displays, such as icons surrounding screen options, to manage the rich visual information. One or a combination of factors may be used to determine the active display, and these may include, among other things, the energy-based (e.g., electrosurgery, ultrasound) or machine-based (e.g., stapler) surgical device in use, the estimated risk associated with a given display, the surgeon's level of experience, and the surgeon's choice. In other embodiments, the visual information may include rich data overlaid or superimposed on the surgical field to manage the visual information. In various embodiments described below, this includes superimposed images that require video analysis and tracking to properly overlay the data. Visual information data thus communicated can provide additional useful visual information to the OR team in a more concise and understandable way, in contrast to static icons.

[0014] In various embodiments, the Disclosure provides techniques for selectively activating on-screen displays, such as icons surrounding a screen, to manage visual information during a surgical procedure. In other embodiments, the Disclosure provides techniques for determining an active display using one or a combination of factors. In various embodiments, the techniques provided by the Disclosure may include, among other things, selecting an energy-based or machine-based surgical device to be used as the active display, estimating the risks associated with a given display, and utilizing the experience level of the surgeon or OR team making the selection.

[0015] In other embodiments, the techniques described herein may include overlaying or superimposing rich data onto the surgical field for the purpose of managing visual information. Several display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Graphical overlays may take the form of transparent graphics, translucent graphics, or opaque graphics, or combinations of transparent, translucent, and opaque elements or effects. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on changes in display priority values. Graphical overlays are rendered on an active display monitor to quickly and efficiently communicate critical information to the OR team.

[0016] In other embodiments, the technology provided by the Disclosure may include superimposing images that require video analysis and tracking in order to appropriately overlay visual information data. In other embodiments, the technology provided by the Disclosure may include communicating rich visual information, as opposed to simple static icons, to provide additional visual information to the OR team in a more concise and easily understandable manner. In other embodiments, the visual overlay may be used in combination with auditory and / or somatosensory overlays, e.g., thermal, chemical, and mechanical devices, and combinations thereof.

[0017] The following description generally pertains to devices, systems, and methods for providing augmented reality (AR) interactive experiences during surgical procedures. In this context, images of the surgical field and surgical instruments and other objects appearing in the surgical field are enhanced by overlaying computer-generated visual, auditory, tactile, somatosensory, olfactory, or other sensory information onto the real-world images of the surgical field, instruments, and / or other objects appearing in the surgical field. The images may be streamed in real time or they may be still images. Augmented reality is a technique for rendering and displaying virtual or "augmented" virtual objects, data, or visual effects that are overlaid on a real environment. The real environment may include the surgical field. Virtual objects overlaid on a real environment may be represented at fixed or set positions relative to one or more aspects of the real environment. In non-limiting examples, if a real-world object moves out of the field of view of the real environment, the virtual object fixed to the real-world object also moves out of the field of view of augmented reality.

[0018] Some of the display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value.

[0019] As described herein, AR is an extended version of the real physical world achieved through the use of digital visual elements, sounds, or other sensory stimuli delivered via technology. Virtual reality (VR) is a computer-generated environment with scenes and objects that appear real, making the user feel immersed in them. This environment is perceived through a device known as a virtual reality headset or helmet. While both mixed reality (MR) and AR are considered immersive technologies, they are not the same. MR is an extension of mixed reality that allows real and virtual elements to interact within an environment. AR often adds digital elements to a live view by using a camera, while an MR experience combines elements of both AR and VR, where real-world and digital objects interact.

[0020] In an AR environment, one or more computer-generated virtual objects may be displayed alongside one or more real-world (i.e., so-called "real-world") elements. For example, real-time images or videos of the surrounding environment may be displayed on a computer screen display along with one or more overlay virtual objects. Such virtual objects can provide supplementary information about the environment or, in general, enhance the user's perception and engagement with the environment. Conversely, real-time images or videos of the surrounding environment can, in addition or alternatively, enhance the user's engagement with the virtual objects displayed on the display.

[0021] Apparatus, systems, and methods in the context of this disclosure enhance images received from one or more imaging devices during a surgical procedure. Imaging devices may include various scopes used during non-invasive and minimally invasive surgical procedures, AR devices, and / or cameras that provide images during incisional surgical procedures. Images may be streamed in real time or still images. Apparatus, systems, and methods provide an augmented reality interactive experience by enhancing images of a real-world surgical environment by overlaying representations of virtual objects or data and / or real objects onto the real-world surgical environment. The augmented reality experience may be viewed on a display and / or AR device that allows the user to view virtual objects overlaid on the real-world surgical environment. The display may be located in the operating room or located away from the operating room. The AR device is worn on the head of a surgeon or other operating room personnel and typically includes two stereoscopic display lenses or screens, one for each eye of the user. Natural light can pass through the two transparent or translucent display lenses so that aspects of the real environment are visible, while projecting light to make virtual objects visible to the user of the AR device.

[0022] Two or more displays and AR devices may be used in conjunction with a first display or AR device that controls one or more additional displays or AR devices in a system having defined roles. For example, when activating a display or AR device, the user may select a role (e.g., a surgeon, surgical assistant, nurse, etc. during a surgical procedure), and the display or AR device may display information related to that role. For example, a surgical assistant may have the display show virtual representations of instruments that the surgeon needs to use for the next step of the surgical procedure. The surgeon's focus on the current step may differ from the information displayed by the surgical assistant.

[0023] While many known on-screen displays and alerts exist, this disclosure provides many novel and unique augmented reality interactive experiences during surgical procedures. Such augmented reality interactive experiences include visual, auditory, tactile, somatosensory, olfactory, or other sensory feedback information to the surgical team inside or outside the operating room. Virtual feedback information overlaid on the real-world surgical environment may be provided to the operating room (OR) team, including, but not limited to, personnel within the OR, such as the surgical surgeon, surgical assistants, scrub wearers, anesthesiologists, and circulating nurses. The virtual feedback information can be communicated on any number of display options, such as primary OR screen displays, AR devices, energy or surgical staplers, tablets, augmented reality glasses, and other devices.

[0024] Figure 1 shows a computer-implemented interactive surgical system 1 comprising one or more surgical systems 2 and a cloud-based system 4. The cloud-based system 4 may include a remote server 13 connected to remote storage 5. Each surgical system 2 comprises at least one surgical hub 6 that communicates with the cloud 4. For example, a surgical system 2 may comprise a visualization system 8, a robotic system 10, and a handheld intelligent surgical instrument 12, each configured to communicate with each other and / or with the hub 6. In some embodiments, a surgical system 2 may comprise M hubs 6, N visualization systems 8, O robotic systems 10, and P handheld intelligent surgical instruments 12, where M, N, O, and P are integers of 1 or more. The computer-implemented interactive surgical system 1 may be configured to provide an augmented reality interactive experience during surgical procedures, as described herein.

[0025] Figure 2 shows an example of a surgical system 2 for performing a surgical procedure on a patient lying on an operating table 14 in a surgical operating room 16. A robotic system 10 is used as part of the surgical system 2 in the surgical procedure. The robotic system 10 includes a surgeon's console 18, a patient-side cart 20 (surgical robot), and a surgical robot hub 22. The patient-side cart 20 allows the surgeon to operate at least one detachably connected surgical tool 17 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 18 or an augmented reality (AR) device 66 worn by the surgeon. Images of the surgical site during the minimally invasive procedure (e.g., still or live images streamed in real time) can be acquired by a medical imaging device 24. The patient-side cart 20 can operate the imaging device 24 to orient it. Images of the incision surgical procedure can be acquired by a medical imaging device 96. The robot hub 22 processes images of the surgical site for subsequent display on the surgeon's console 18, or on an AR device 66 worn by the surgeon or another person in the surgical room 16.

[0026] The optical components of the imaging device 24, 96, or AR device 66 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from tissues and instruments in the surgical field.

[0027] In various embodiments, the imaging device 24 is configured for use in minimally invasive surgical procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes. In various embodiments, the imaging device 96 is configured for use in incisional (invasive) surgical procedures.

[0028] In various embodiments, the visualization system 8 comprises one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing devices, one or more storage arrays, and one or more displays. In one embodiment, the visualization system 8 includes interfaces for HL7, PACS, and EMR. In one embodiment, the imaging device 24 may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range in the electromagnetic spectrum. Wavelengths are separated by filters or by instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging can extract information invisible to the human eye. Multispectral monitoring allows the surgical field to be repositioned after the surgical task for performing tests on the treated tissue is completed.

[0029] Figure 2 shows a primary display 19 positioned in the sterile field for the operator on the operating table 14 to see. The visualization tower 11 includes a first non-sterile display 7 and a second non-sterile display 9 positioned outside the sterile field and facing opposite directions from each other. The visualization system 8, guided by the hub 6, is configured to utilize displays 7, 9, and 19 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 6 can cause the visualization system 8 to display AR images of the surgical site recorded by imaging devices 24 and 96 through the non-sterile displays 7, 9, or AR device 66, while maintaining live video of the surgical site on the primary display 19 or AR device 66. The non-sterile displays 7 and 9 can, for example, enable non-sterile operators to perform diagnostic steps related to the surgical procedure.

[0030] Figure 3 shows a hub 6 that communicates with a visualization system 8, a robotic system 10, and handheld intelligent surgical instruments 12. The hub 6 includes a hub display 35, an imaging module 38, a generator module 40, a communication module 30, a processor module 32, a storage array 34, and an operating room mapping module 33. The hub 6 further includes a smoke extraction module 26 and / or a suction / irrigation module 28. In various embodiments, the imaging module 38 includes an AR device 66, and the processor module 32 includes an integrated video processor and an augmented reality modeler (e.g., as shown in Figure 10). Modular light sources can be adapted for use with various imaging devices. In various examples, multiple imaging devices can be positioned at different locations in the surgical field to provide multiple views (e.g., non-invasive, minimally invasive, invasive, or incisional surgical procedures). The imaging module 38 can be configured to switch between imaging devices to provide the optimal view. In various embodiments, the imaging module 38 can be configured to integrate images from different imaging devices and provide an augmented reality interactive experience during surgical procedures as described herein.

[0031] Figure 4 shows a surgical data network 51 including a modular communication hub 53 configured to connect modular devices located in one or more operating rooms / surgery sites of a medical facility to a cloud-based system. The cloud 54 may include a remote server 63 (Figure 5) connected to a storage device 55. The modular communication hub 53 includes a network hub 57 and / or a network switch 59 that communicate with a network router 61. The modular communication hub 53 is connected to a local computer system 60 for data processing. The operating room modular devices 1a-1n may be connected to the modular communication hub 53. The network hub 57 and / or the network switch 59 are connected to the network router 61 so that devices 1a-1n can connect to the cloud 54 or the local computer system 60. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. The operating room devices 1a-1n may be connected to the modular communication hub 53 via a wired channel or a wireless channel. The surgical data network environment 51 may be employed, as described herein, to provide an augmented reality interactive experience during a surgical procedure, in particular to provide augmented images of the surgical field to one or more remote displays 58.

[0032] Figure 5 shows a computer-implemented interactive surgical system 50. The computer-implemented interactive surgical system 50 is similar in many respects to the computer-implemented interactive surgical system 1. The computer-implemented interactive surgical system 50 includes one or more surgical systems 52 that are similar in many respects to surgical system 2. Each surgical system 52 includes at least one surgical hub 56 that communicates with a cloud 54 which may include a remote server 63. In one embodiment, the computer-implemented interactive surgical system 50 includes a modular control tower 23 connected to a plurality of surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 23 includes a modular communication hub 53 connected to a computer system 60.

[0033] Returning to Figure 5, the modular control tower 23 is connected to an imaging module 38 connected to an endoscope 98, a generator module 27 connected to an energy device 99, a fume exhaust module 76, a suction / irrigation module 78, a communication module 13, a processor module 15, a storage array 16, and optionally smart devices / instruments 21 and sensor modules 29 connected to a display 39. Surgical site devices are connected to cloud computing resources such as a server 63, data storage 55, and a display 58 via the modular control tower 23. The robot hub 72 may also be connected to the modular control tower 23, as well as the server 63, data storage 55, and display 58. In particular, the devices / instruments 21 and the visualization system 58 may be connected to the modular control tower 23 via wired or wireless communication standards or protocols as described herein. The modular control tower 23 may be connected to a hub display 65 (e.g., a monitor, screen) to display received augmented images, including overlaid virtual objects on the real surgical world, received from the imaging module 38, device / instrument display 39, and / or other visualization systems 58. The hub display 65 may also display data received from devices connected to the modular control tower 23, along with the images and overlay images.

[0034] Figure 6 shows a surgical hub 56 including multiple modules connected to a modular control tower 23. The modular control tower 23 includes a modular communication hub 53, such as a network connectivity device, and a computer system 60 for local processing, visualization, and imaging of augmented surgical information. The modular communication hub 53 is connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 53, and data associated with the modules may be transferred to the computer system 60, cloud computing resources, or both. Each of the network hubs / switches 57 / 59 within the modular communication hub 53 may include three downstream ports and one upstream port. The upstream network hubs / switches 57, 59 are connected to the processor 31 to provide communication connectivity to cloud computing resources and local displays 67. Communication to the cloud 54 can be done via either a wired communication channel or a wireless communication channel.

[0035] The computer system 60 includes a processor 31 and a network interface 37. The processor 31 is connected via a system bus to a communication module 41, storage 45, memory 46, non-volatile memory 47, and an input / output interface 48. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using various available bus architectures.

[0036] The processor 31 may include an augmented reality modeler (e.g., as shown in Figure 10) and may be implemented as a single-core or multi-core processor, such as one known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0037] System memory includes volatile and non-volatile memory. The Basic Input / Output System (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory may include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0038] The computer system 60 also includes removable / non-removable volatile / non-volatile computer storage media, such as disk storage devices. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0039] In various embodiments, the computer system 60 in Figure 6, the imaging module 38 in Figures 4 to 6, and / or the visualization system 58, and / or the processor module 15 may include an image processor, an image processing engine, an image processing unit (GPU), a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0040] Figure 7 shows an augmented reality (AR) system 263 that includes an intermediate signal coupler 64 located in the communication path between an imaging module 38 and a surgical hub display 67. The signal coupler 64 combines audio and / or image data received from the imaging module 38 and / or an AR device 66. The surgical hub 56 receives the combined data from the coupler 64, overlays the provided data onto the display 67, and displays the overlaid data. The imaging device 68 may be a digital video camera, and the audio device 69 may be a microphone. The signal coupler 64 may include a wireless head-up display adapter for coupling to an AR device 66 located in the communication path of the display 67 to a console that enables the surgical hub 56 to overlay data onto the display 67.

[0041] Figure 8 shows an augmented reality (AR) system including an intermediate signal coupler positioned in the communication path between the imaging module and the surgical hub display. Figure 8 shows an AR device 66 worn by the surgeon 73 to communicate data to the surgical hub 56. Peripheral information of the AR device 66 does not include active images. Rather, peripheral information includes only signals that do not have the same requirements for device settings or refresh rate. The interaction may extend the surgeon 73's information based on links with preoperative computed tomography (CT) or other data linked within the surgical hub 56. The AR device 66 can identify structures and, for example, ask whether an instrument is touching a nerve, blood vessel, or adhesion. The AR device 66 may include processing in the surgical hub 56 used to provide preoperative scan data, optical views, tissue examination characteristics acquired throughout the procedure, and / or answers. The surgeon 73 may write notes on the AR device 66 so that they are stored in the hub storage 45 along with patient data for later use in reporting or follow-up.

[0042] The AR device 66, worn by the surgeon 73, links to the surgical hub 56 using auditory and visual information to avoid the need for overlays, and allows customization of information displayed around the periphery of the field of view. The AR device 66 provides signals from devices (e.g., instruments) and answers queries regarding location information linked with video to identify device settings or quadrants or locations. The AR device 66 has voice control and voice feedback from the AR device 66. The AR device 66 can interact with other systems in the operating room and can have available feedback and interaction wherever the surgeon 73 looks. For example, the AR device 66 may receive voice or gesture start commands and queries from the surgeon, and the AR device 66 may provide feedback in the form of one or more modalities, including voice, visual, or haptic touch.

[0043] Figure 9 shows a surgeon 73 and a patient 74 wearing AR devices 66, and may include a camera 96 ​​in the operating room 75. The AR device 66 worn by the surgeon 73 may be used to present virtual objects overlaid on a real-time image of the surgical field to the surgeon 73 through an augmented reality display 89 or through a hub-connected display 67. The real-time image may include parts of surgical instruments 77. The virtual objects may not be visible to others in the operating room 75 (e.g., surgical assistants or nurses), but they may also wear AR devices 66. Even if another person is viewing the operating room 75 using an AR device 66, that person may not be able to see the virtual objects, or may be able to see the virtual objects in augmented reality shared with the surgeon 73, or may be able to see modified versions of the virtual objects (e.g., according to customization specific to the surgeon 73), or may see different virtual objects.

[0044] Virtual objects and / or data may be configured to appear on a portion of the surgical instrument 77 or within the surgical field captured by the imaging module 38, the imaging device 68 during minimally invasive surgical procedures, and / or the camera 96 ​​during incisional surgical procedures. In the illustrated example, the imaging module 38 is a laparoscopic camera that provides live images of the surgical area during minimally invasive surgical procedures. The AR system may present virtual objects fixed to real objects regardless of the viewpoint of one or more viewers of the AR system (e.g., the surgeon 73). For example, virtual objects may be visible to viewers of the AR system inside the operating room 75, but not visible to viewers of the AR system outside the operating room 75. Virtual objects may be displayed to viewers outside the operating room 75 when a viewer enters the operating room 75. Augmented images may be displayed on the surgical hub display 67 or the augmented reality display 89.

[0045] The AR device 66 may include one or more screens or lenses, such as a single screen or two screens (e.g., one for each user's eye). The screens may allow light to pass through them so that aspects of the real environment are visible while virtual objects are being displayed. Virtual objects may become visible to the surgeon 73 by projecting light. Virtual objects may appear to have some degree of transparency or may be opaque (i.e., blocking aspects of the real environment).

[0046] An AR system may be visible to one or more viewers and may include differences between views available to one or more viewers, while maintaining some common aspects between views. For example, a heads-up display may change between two views, but virtual objects and / or data may be fixed to real objects or areas in both views. Aspects such as the color, lighting, or other changes of an object may occur between views without changing the fixed position of at least one virtual object.

[0047] Users can view virtual objects and / or data presented within the AR system as opaque or with a certain level of transparency. For example, a user can interact with a virtual object by moving it from a first position to a second position. For instance, a user may move an object with their own hand. This may be done virtually in the AR system by determining that the hand has moved to a position adjacent to or adjacent to the object (using one or more cameras, which may be mounted on the AR device 66, such as AR device camera 79 or a separate camera 96, and which may be static or controlled to move) and moving the object accordingly. The virtual form may include a virtual representation of a real-world object, or it may include visual effects such as lighting effects. The AR system may include rules to govern the behavior of the virtual object, such as exposing the virtual object to gravity or friction, or it may include other predefined rules that negate real-world physical constraints (e.g., floating objects, perpetual motion, etc.). The AR device 66 may include a camera 79 (which should not be confused with a separate camera 96). The AR device camera 79 or camera 96 ​​may include an infrared camera, an infrared filter, a visible light filter, multiple cameras, a depth camera, etc. The AR device 66 may project virtual items onto a representation of the real environment that the user can see.

[0048] The AR device 66 may be used, for example, in an operating room 75 during a surgical procedure performed on a patient 74 by a surgeon 73. The AR device 66 may project or display virtual objects, such as virtual objects during the surgical procedure, to extend the surgeon's vision. The surgeon 73 may view the virtual objects using the AR device 66, a remote controller for the AR device 66, or interact with the virtual objects by using his hands to “interact” with the virtual objects or gestures recognized by the camera 79 of the AR device 66, for example. The virtual objects can extend surgical tools, such as surgical instruments 77. For example, the virtual object may appear to be connected to the surgical instrument 77 (to the surgeon 73 viewing the virtual object through the AR device 66), or to remain at a fixed distance from the surgical instrument 77. In another example, the virtual object may be used to guide the surgical instrument 77 and may appear to be fixed to the patient 74. In certain examples, the virtual object may react to the movement of other virtual or real-world objects in the surgical field. For example, a virtual object may be modified when a surgeon is manipulating a surgical instrument in close proximity to the virtual object.

[0049] The augmented reality display system imaging device 38 captures real images of the surgical area during the surgical procedure. The augmented reality displays 89 and 67 present an overlay of the operating modes of the surgical instrument 77 onto the real images of the surgical area. The surgical instrument 77 includes a communication circuit 231 for communicating operating mode and functional data from the surgical instrument 77 to the AR device 66 via a communication circuit 233 on the AR device 66. The surgical instrument 77 and the AR device 66 are shown in RF wireless communication between circuits 231 and 233, as indicated by arrows B and C, but other communication technologies (e.g., wired, ultrasonic, infrared, etc.) may be employed. The overlay relates to the operating modes of the surgical instrument 77 that are actively visualized. The overlay combines the modes of tissue interaction in the surgical area with functional data from the surgical instrument 77. The processor portion of the AR device 66 is configured to receive operating mode and functional data from the surgical instrument 77, determine overlays related to the operation of the surgical instrument 77, and combine the tissue characteristics within the surgical area with the functional data from the surgical instrument 77. The augmented images display alerts regarding device performance considerations, non-conformity use, and incomplete capture. Non-conformity use includes out-of-range tissue conditions and improperly balanced tissue within the jaws of the end effector. Additional augmented images provide displays of incidental events, including tissue tension and foreign body detection. Other augmented images display device status overlays and instrument displays.

[0050] Figure 10 shows a system 83 for augmenting images of the surgical field with information using an AR display 89, according to at least one aspect of the present disclosure. The system 83 may be used to perform the techniques described below, for example, by using a processor 85. The system 83 includes one aspect of an AR device 66 that can communicate with a database 93. The AR device 66 includes a processor 85, memory 87, an AR display 89, and a camera 79. The AR device 66 may also include a sensor 90, a speaker 91, and / or a haptic controller 92. The database 93 may include image storage 94 or preoperative planning storage 95.

[0051] The processor 85 of the AR device 66 includes an augmented reality modeler 86. The augmented reality modeler 86 may be used by the processor 85 to create an augmented reality environment. For example, the augmented reality modeler 86 may receive images of instruments in the surgical field from a camera 79 or sensor 90, etc., and create an augmented reality environment that fits within the displayed image of the surgical field. In another example, physical objects and / or data may be overlaid on the surgical field and / or surgical instrument images, and the augmented reality modeler 86 may use the physical objects and data to present an augmented reality display of virtual objects and / or data within the augmented reality environment. For example, the augmented reality modeler 86 may use or detect instruments at the patient's surgical site and present virtual objects and / or data on the surgical instruments, and / or images of the surgical site in the surgical field captured by the camera 79. The AR display 89 may display the AR environment overlaid on the real environment. The display 89 can use the AR device 66, which is located in a fixed position, etc., within the AR environment, to show virtual objects and / or data.

[0052] The AR device 66 may include sensors 90 such as infrared sensors. The camera 79 or sensor 90 may be used to detect movements such as gestures by a surgeon or other user, which may be interpreted by the processor 85 as attempted or intended interactions by the user with a virtual target. The processor 85 can identify objects in the real environment by processing information received using the camera 79, for example. In other embodiments, sensor 90 may be a tactile sensor, audible sensor, chemical sensor, or thermal sensor to generate corresponding signals that can be combined with various data feeds to create an augmented environment. Sensor 90 may include binaural audio sensors (spatial sound), inertial measurement (accelerometer, gyroscope, magnetometer) sensors, environmental sensors, depth camera sensors, hand and eye-tracking sensors, and voice command recognition capabilities.

[0053] The AR display 89 may, for example, during a surgical procedure, allow the surgical field to be viewed through the AR display 89, while presenting virtual features within the surgical field that correspond to physical features hidden by the patient's anatomical features. The virtual features may have a virtual position or orientation that corresponds to a first physical position or orientation of the physical features. In one example, the virtual position or orientation of the virtual features may include an offset from the first physical position or orientation of the physical features. The offset may include a predetermined distance from the augmented reality display, a relative distance from the augmented reality display to the anatomical features, and so on.

[0054] In one example, the AR device 66 may be an individual AR device. In one embodiment, the AR device 66 may be a HoloLens 2 AR device manufactured by Microsoft in Redmond, Washington. This AR device 66 includes a visor with lenses and binaural audio features (spatial sound), inertial measurements (accelerometer, gyroscope, magnetometer), environmental sensors, a depth camera, a video camera, hand and eye tracking, and voice command recognition capabilities. It provides a high-resolution, improved field of view by using mirrors to orient waveguides in front of the wearer's eyes. The image can be magnified by changing the angle of the mirrors. It also provides eye tracking to recognize the user and adjust the lens width for a particular user.

[0055] In another example, AR device 66 could be the Snapchat Spectacles 3 AR device. This AR device offers the ability to capture paired images, recreate 3D depth mapping, add virtual effects, and play 3D videos. The AR device includes two HD cameras for capturing 3D photos and videos at 60fps, while four built-in microphones record immersive high-fidelity audio. Images from both cameras are combined to construct a geometric map of the real world around the user, providing a new sense of depth perception. Photos and videos can be wirelessly synchronized to an external display device.

[0056] In yet another example, AR device 66 could be Google's Glass 2 AR device. This AR device provides inertial measurement (accelerometer, gyroscope, magnetometer) information overlaid on the lens (outside the field of view) to supplement the information.

[0057] In another example, AR device 66 could be an Echo Frames AR device from Amazon. This AR device does not have a camera / display. The microphone and speaker are linked to Alexa. This AR device has fewer features than a head-up display.

[0058] In yet another example, AR device 66 could be the Focals AR device by North (Google). This AR device provides a notification pusher / smartwatch analog, inertial measurement, screen overlays for information (weather, calendar, messages), and voice control (Alexa) integration. This AR device also provides basic head-up display functionality.

[0059] In another example, AR device 66 could be an Nreal AR device. This AR device includes spatial sound, two ambient cameras, a photographic camera, an IMU (accelerometer, gyroscope), an ambient light sensor, and proximity sensor functions. Nebula projects application information onto the lens.

[0060] In various other examples, the AR device 66 may be any one of the following commercially available AR devices, namely Magic Leap 1, Epson Moverio, Vuzix Blade AR, ZenFone AR, Microsoft AR glasses prototype, or EyeTap, which create light collinear with the ambient light directly onto the retina. A beam splitter makes the same visible light available to a computer, for example, to process and overlay information. The AR visualization system may include a HUD, contact lenses, glasses, virtual reality (VR) headset, virtual retinal display, intraoperative display, and / or smart contact lenses (bionic lenses).

[0061] The multi-user interface for the AR device 66 includes a virtual retinal display such as a raster display that draws directly onto the retina rather than on a screen in front of the eyes, a smart TV, a smartphone, and / or a spatial display such as the Sony Spatial Display System.

[0062] Other AR technologies may include, for example, AR capture devices and software applications, AR creation devices and software applications, and AR cloud devices and software applications. AR capture devices and software applications include, for example, the Apple Polycam app and Ubiquity 6 (Mirrorworld using the Display.land app), which allow users to scan and acquire 3D images of the real world (to create 3D models). AR creation devices and software applications include, for example, Adobe Aero, Vuforia, ARToolKit, Google ARCore, Apple ARKit, MAXST, Aurasma, Zappar, and Blippar. AR cloud devices and software applications include, for example, Facebook, Google (world geometry, object recognition, predictive data), Amazon AR Cloud (commerce), Microsoft Azure, Samsung Project Whare, Niantic, and Magic Leap.

[0063] Situational awareness is the ability of several embodiments of a surgical system to determine or infer information related to a surgical procedure from data received from a database and / or instruments. This information may include the type of procedure being performed, the type of tissue being operated on, or the body cavity being treated. Contextual information regarding a surgical procedure can improve a surgical system, for example, by controlling modular devices connected to it (e.g., robotic arms and / or robotic surgical tools) and providing contextualized information or suggestions to the surgeon during the course of the surgical procedure.

[0064] Figure 11 shows a timeline of a situation-aware surgical procedure. Figure 11 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each stage of the surgical procedure. Timeline 5200 shows the typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy procedure, which begins with setting up the operating room and ends with transferring the patient to the postoperative recovery room. Throughout the surgical procedure, the situation-aware surgical hub 5104 receives data from data source 5126, including data generated each time healthcare professionals use the modular device 5102 paired with the surgical hub 5104. By receiving this data from the paired modular device 5102 and other data sources 5126, the surgical hub 5104 can continuously derive estimations (i.e., contextual information) about the ongoing procedure as new data is received, such as which stage of the procedure is being performed at any given time. The situation awareness system of the surgical hub 5104 can, for example, record data relating to a procedure to generate a report, verify the steps being taken by a healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as described above.

[0065] In the first step 5202, hospital staff retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery.

[0066] In the second 5204, staff scan incoming medical supplies for a procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies used in various types of procedures to confirm that the mixture of supplies corresponds to a thoracic procedure. Furthermore, the surgical hub 5104 can also determine that the procedure is not a wedge resection (because the incoming supplies either do not contain specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection).

[0067] In the third 5206, a healthcare worker scans the patient band via a scanner 5128 that is communicably connected to a surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data.

[0068] In the fourth part of 5208, a medical professional turns on the assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they include a fume exhauster, a ventilator, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104 located within a specific vicinity of the modular device 5102 as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 determines that the surgical procedure is a VATS surgery based on this particular combination of paired modular devices 5102. Based on the combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team. Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing.

[0069] In step 5210, the staff attaches the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, the surgical hub 5104 confirms that the patient is in the operating room.

[0070] In step 6, 5212, medical personnel induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from modular devices 5102 and / or patient monitoring devices 5124, including, for example, EKG data, blood pressure data, ventilator data or a combination thereof. Once step 6, 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion commences.

[0071] In section 7 of 5214, the lung of the patient being operated on collapses (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer from the ventilator data that the patient's lung has collapsed. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has begun and determine that causing the lung to collapse is the first surgical step in this particular procedure.

[0072] In step 8, 5216, a medical imaging device 5108 (e.g., a scope) is inserted, and video footage from the medical imaging device is initiated. The surgical hub 5104 receives medical imaging device data (i.e., still image data or real-time live-streaming video) through its connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has commenced. Furthermore, the surgical hub 5104 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resection has not already been taken into consideration by the surgical hub 5104 based on the data received in step 2, 5204 of the procedure). Using data from the medical imaging device 124 (Figure 2), contextual information regarding the type of procedure being performed can be determined in various ways, for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number of medical imaging devices being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used.

[0073] For example, one technique for performing VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing VATS segmentectomy positions the camera in an anterior intercostal position relative to the segmental fissure. The situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy utilizes a single medical imaging device, while another technique for performing VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0074] In Section 9, 5218, the surgical team initiates the incision step of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the steps of the procedure described above) corresponds to the incision step.

[0075] In step 10, 5220, the surgical team proceeds to the ligation step of the procedure. The surgical hub 5104 receives data from the surgical stapling and cutting instruments indicating that the instruments are being fired, so it can infer that the surgeon is ligating the arteries and veins. Similar to the previous step, the surgical hub 5104 can derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps in the read process.

[0076] In Section 11, 5222, the segmental resection portion of the procedure is performed. The surgical hub 5104 infers that the surgeon is transversely incising parenchymal tissue based on data from surgical instruments, including data from a staple cartridge. Cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Cartridge data may indicate the type of tissue being stapled and / or transversely incised for different types of staples used for different types of tissue. The type of staples fired is used for parenchymal tissue or other similar tissue types, and the surgical hub 5104 can infer that a segmental resection procedure is being performed.

[0077] Next, in step 12, 5224, the nodule incision process is performed. Based on the data received from the generator indicating that an RF or ultrasonic instrument is being emitted, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision process, thereby enabling the surgical hub 5104 to make this inference. Note that the surgeon will periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific step in the procedure, as different instruments are better suited to specific tasks. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once step 12, 5224 is completed, the incision is closed and the postoperative portion of the procedure begins.

[0078] In step 13, 5226, the patient is de-anesthetized. The surgical hub 5104 can estimate that the patient is waking from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase).

[0079] Finally, in the 14th step 5228, the healthcare worker removes the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 can infer that the patient has been transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. Based on the data received from various data sources 5126 that are communicably connected to the surgical hub 5104, the surgical hub 5104 can determine or infer when each step of a given surgical procedure is occurring.

[0080] As shown in the first step 5202 of the timeline 5200 shown in Figure 86, in addition to using patient data from the EMR database(s) to estimate the type of surgical procedure to be performed, the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0081] Surgical displays (e.g., displays 7, 9, 19, 35, 62, 65, 66, 67, and 89) play a vital role in the operating room by providing useful information to clinicians (e.g., surgeons, surgical staff), which can be used, among other things, to assess the progress of surgical procedures, determine subsequent steps for performing surgical procedures, and monitor the patient's vital signs. The displays need to be large enough to see the information provided, but not excessively large, so as not to hinder workflow or movement in a crowded operating room.

[0082] For example, an imaging device, such as one of the many imaging devices described elsewhere in this specification, is used to capture a live stream of the surgical field during a surgical procedure. A display shows this live stream captured by the imaging device so that the clinician can view the surgical field during the surgical procedure.

[0083] During a surgical procedure, information related to or associated with the procedure may be overlaid on a live stream on a display. For example, an electrocardiogram (EKG) may monitor the patient's heart rate during the procedure, and the monitored heart rate may be overlaid on the live stream to ensure that the clinician can assure the patient that the patient is stable.

[0084] Various other sensors, detectors, and modules can monitor other parameters throughout the surgical procedure, and information associated with these parameters can also be overlaid on the display. However, some overlaid information may be more important than others. For example, when a clinician is manipulating tissue using the end effector of a surgical instrument, information about the amount of force being applied to the tissue using the end effector is relevant to monitoring to ensure that the tissue is not unintentionally damaged.

[0085] However, due to the sheer volume of information overlaid on the display, more important information, such as the forces being applied to tissue, may be overlooked or missed by clinicians. This abundance of competing information can overwhelm surgeons with information that may be detrimental to their ability to perform surgical procedures properly and may prove costly for the patient. Therefore, the amount of data overlaid on the display needs to be prioritized, controlled, and / or limited.

[0086] Figure 12 shows a surgical visualization system 6000 according to at least one aspect of the present disclosure. Various components of the surgical visualization system 6000 are in many respects similar to components of other systems described elsewhere in the present disclosure and are therefore not repeated herein at the same level of detail for the sake of brevity. In some embodiments, the system 6000 is a standalone system. In other embodiments, the system 6000 is integrated into or used in conjunction with a computer-implemented interactive surgical system 1.

[0087] The surgical visualization system 6000 includes a control module 6001 configured to perform various techniques described herein by using one or more processors or processing circuits, such as a processor 85. In some embodiments, the system 6000 may include, be used with, or communicate with, an augmented reality device 84, for example. The system 6000 may further include a storage medium, such as a memory 6003, an imaging device 6004, such as a camera 88, and a display 6005. The system 6000 may further include one or more speakers 91, a tactile controller 92, and / or sensors 90 (see Figure 10). The display 6005 may include, for example, an AR display 89, a VR display, a projector, a head-up display, a screen, and / or any other suitable device for displaying visual content.

[0088] In some embodiments, system 6000 is incorporated, for example, into a computer-implemented interactive surgical system 50. In some embodiments, system 6000 operably communicates with one or more hubs, systems, networks, servers, and / or databases that can deliver surgical data to system 6000. For example, system 6000 may operably communicate with a cloud 54, which may include a remote server 63, a robot hub 72, a surgical hub 56, devices / instruments 21, and / or a modular control tower 23, via wired or wireless communication standards or protocols as described herein. In some embodiments, system 6000 includes a context-aware module 6006 similar to that described in relation to the surgical hub 5104. The context-aware module 6006 may be trained to extrapolate contextual information about surgical procedures based on a large amount of perioperative data received through sensor inputs and / or user inputs.

[0089] Figure 13 is a flowchart illustrating the operation of an exemplary method 6010 for determining the display placement of competing surgical data for presentation on a display such as display 6005. Method 6010 includes detecting surgical data (6011), assigning a display priority value or display priority status to the surgical data (6012), and determining the display placement of the surgical data on the display based on the display priority value (6013). Method 6010 may further include presenting the surgical data on a live stream of the surgical field, for example, by displaying or overlaying a visual representation of the surgical data according to the display placement (6014).

[0090] In some embodiments, surgical data is detected by the control module 6001 (6011). Surgical data can be detected by receiving surgical data from one or more sources, such as components of the computer-implemented interactive surgical system 1, via one or more wireless and / or wired communication interfaces (6011). In at least one example, the surgical data may include data received from one or more surgical instruments 21. In another example, the surgical data includes contextual information confirmed by the situation awareness module 6006.

[0091] In specific examples, surgical data may include control data, biomarker measurements, and / or other operational indicators of operation and / or results associated with the surgical instrument 21. In specific examples, surgical data may be any data indicating a higher tendency for malformed staples and poorly sealed tissue. In specific examples, surgical data may be associated with tissue flow, clamping force, and firing force, among other tissue and / or instrument parameters, which can be monitored and displayed to the clinician in real time in multiple ways to allow adjustment of the firing process or to alert the surgeon of potentially malformed staple areas.

[0092] In some embodiments, the display priority value is assigned based on surgical data and / or contextual information relating to a surgical procedure developed by the situation awareness module 6006. In some embodiments, the display priority value is assigned based on a trigger event, condition, or characteristic of the surgical data. In some embodiments, assigning a display priority value (6012) includes changing a previously assigned display priority value. For example, detection of a trigger event, condition, and / or characteristic of the surgical data may change a previously assigned display priority value to a higher or lower value.

[0093] In a specific example, processor 85 employs a predetermined equation and / or formula when determining the display priority value of surgical data. Various relevant factors may be considered and different weights assigned when calculating the display priority value. In addition or alternatively, one or more databases or tables listing surgical data and their corresponding display priority values ​​may be used by processor 85 when assigning the display priority value.

[0094] In various embodiments, the assigned (6012) display priority value includes various levels of display priority, such as a low display priority level, a medium display priority level, and / or a high display priority level. In some embodiments, the display priority value is a display priority status, such as a high priority status, a medium priority status, and / or a low priority status.

[0095] Figure 14 is a flowchart illustrating the operation of an exemplary method 6020 for determining the display priority value of detected (6011) surgical data according to method 6010 of Figure 13. In some embodiments, the display priority value depends on the surgical data. In the illustrated example, the display priority value is assigned based on the proximity of the surgical instrument used in the surgical procedure to a significant anatomical structure associated with the surgical procedure. The display priority value is based on the relationship between the received proximity data and a predetermined proximity threshold. For example, if the distance between the surgical instrument and the anatomical structure is greater than a predetermined threshold (6021), the proximity data is assigned a low display priority value (6022). However, if the distance is less than or equal to a predetermined proximity threshold (6021), the proximity data is assigned a high display priority value (6023).

[0096] In some embodiments, the system 6000 employs a situation recognition module 6006 to identify the type of surgical procedure to be performed. The type of surgical procedure may be determined, for example, from user input. Alternatively or in addition, it may be determined from an inventory list of devices selected for use with the surgical procedure, which are specific to or characteristic of the surgical procedure type. The system 6000 may further identify critical structures associated with the surgical procedure, for example, from a database and / or user input. In some embodiments, the system 6000 can detect critical structures in a live stream of the surgical field, such as those captured by an imaging device. Furthermore, the system 6000 can further detect surgical instruments 21 in the surgical field and track the proximity of the surgical instruments 21 to critical structures. The display priority value of the proximity data can be determined as described in relation to Figure 14.

[0097] In some embodiments, the identification of critical structures and / or surgical instruments within a live stream of the surgical field can be achieved through various appropriate object recognition, object tracking, object labeling, and / or other image processing techniques, such as those discussed in U.S. Patent Application Publication No. END9228USNP1 / 190580-1M, titled "STRUCTURED MULTI SPECTRAL COMPUTATIONAL ANALYSIS," which are incorporated by reference as a whole. For example, surgical instruments and / or critical structures in the surgical field can be identified by utilizing previously stored images of the surgical instruments and / or critical structures.

[0098] Low anterior resection (LAR) surgical procedure is a common surgical procedure for rectal cancer. This procedure involves the removal of the rectum. The colon is then attached to the remainder of the rectum to allow normal bowel movement. A circular stapler is commonly used in low LAR procedures. Initially, when the surgeon begins setting up the structure to create the anastomosis, certain parameters, such as tissue tension and anastomotic tissue pressure, can be distracting if they are overlaid or highlighted too early on the live stream and are not relevant. In certain cases, to avoid clutter and / or reduction of the display space available for the live stream, such parameters are overlaid and / or highlighted on display 6005 for each display arrangement according to method 6010.

[0099] In some embodiments, display priority values ​​are assigned to the parameters of tissue tension and anastomotic tissue pressure based on trigger events associated with the relevance of the parameters to the surgical procedure. The trigger event may be, for example, the detection of the connection of the anvil of a circular stapler to a circular stapler trocar. Detection may be achieved automatically, for example, by employing one or more object recognition, object tracking, object labeling, and / or other image processing algorithms of a live stream, and / or through one or more sensors in the anvil and / or trocar that are triggered by the connection or proximity of the anvil to the trocar.

[0100] In some embodiments, trigger events are associated with an increase in severity or risk level. In certain examples, a trigger event may result in a warning and / or immediate suspension of surgical activity, such as suspending staple firing of a surgical instrument 21. A trigger event may result in a transition to a suspended failure mode, for example, in which a series of commands are provided to repair or mitigate the cause of the failure. As will be described in more detail below, trigger events may include, for example, buttress plowing, tissue cutting without tissue sealing, and / or a destroyed anvil. In some embodiments, these trigger events are automatically and visually detected, for example, through object recognition, object tracking, object labeling, and / or other suitable image processing techniques for live stream image frames, or through various suitable wired and / or wireless communication methods.

[0101] In some embodiments, the failure mode is caused by buttress tillage, a condition that can occur when the buttress is used in tissue stapling by the surgical instrument 21. In response to the detection of buttress tillage, the control module 6001 causes the surgical instrument 21 to stop its firing sequence, for example. For example, the control module 6001 may communicate the firing stop command to the surgical instrument 21 via a wireless or wired interface. In addition, the control module 6001 may display or overlay a series of warnings and / or commands to correct the failure by applying tension to the tissue during firing, for example, on a live stream of the surgical field.

[0102] Alternatively, this failure may be caused by detecting tissue cutting without tissue sealing. For example, the control module 6001 may detect a failure of a staple deployed in the tissue grasped by the end effector of the surgical instrument 21 as the cutting member of the surgical instrument 21 advances, which results in tissue cutting without tissue sealing failure. In response to detecting a failure, the control module 6001 may display or overlay a warning and / or a series of commands to repair the failure on the live stream of the surgical field. The commands may suggest clamping the ambient blood supply and preparing material to stop bleeding before releasing the tissue from the jaws of the end effector of the surgical instrument 21.

[0103] Figure 15 is a flowchart illustrating the operation of an exemplary method 6030 for determining display priority values ​​for tissue tension and / or pressure parameters within a surgical anastomosis. Method 6030 includes receiving a tissue parameter (6031) and assigning a display priority value to the parameter based on a trigger event, such as detection of a connection between the anvil and trocar of a circular stapler. For example, if no trocar and anvil connection is detected (6032), a low display priority value is assigned to the parameter (6033). However, if no trocar and anvil connection is detected (6032), a high display priority value is assigned to the parameter (6034).

[0104] Method 6030 provides an example of using the detection of connections of circular staple components as a trigger event for determining a display priority value, but connections of other components of other instruments 21 can be used as trigger events for determining a display priority value. For example, cartridge reloading, end effector mounting, and / or shaft mounting can represent trigger events for determining a display priority value. In some embodiments, surgical instrument components, surgical robot components, and / or assemblies of any suitable surgical system can be used as trigger events for determining a display priority value.

[0105] Figure 16 is a flowchart illustrating the operation of an exemplary method 6040 for determining display priority values ​​based on trigger events. In the illustrated example, the trigger event is the activation of the surgical instrument 21 before receiving parameters necessary to adjust the settings of the surgical instrument 21 for optimal operation of the surgical instrument 21. In some embodiments, the system 6000 may be configured to detect the surgical instrument 21 in a live stream of the surgical field and await user input of the necessary parameters.

[0106] In some embodiments, parameters may be required user inputs. Parameters may be associated with tissue characteristics or disease conditions. Based on the tissue condition and / or disease condition, specific device settings may be adjusted before utilizing the device to treat the tissue. These adjustments may include reducing the firing rate of surgical stapling instruments to ensure a more secure seal. In the case of surgical energy devices, the surgeon may adjust the power in response to new tissue characteristics, for example, to provide a better seal of the tissue.

[0107] As shown in Figure 16, method 6040 includes detecting an attempt by the user to activate the surgical instrument 21 (6041). If the required parameters are received (6042), a low display priority value is assigned (6043). However, if the trocar and anvil connections are not detected (6042), a high display priority value is assigned to the parameters (6044).

[0108] In some embodiments, the parameter is a sensor parameter, which can be an internal sensor of the surgical instrument 21 or any other sensor configured to measure parameters necessary for the proper operation of a surgical procedure. Detection of a trigger event, such as the activation of the surgical instrument 21 before receiving the parameter, may cause the system 6000 to assign a high priority value to the visual content, for example in the form of an overlay, to ignore the missing parameter, request permission to proceed without the missing parameter, or, for example, request input of the missing parameter.

[0109] In some embodiments, the trigger event is a sensor parameter that deviates from a predetermined acceptable range or threshold. The sensor parameter may be, for example, a tissue impedance parameter measurable by a surgical instrument grasping tissue in the surgical field by performing impedance spectroscopy. If the grasped tissue is highly saturated with saline solution, the measured tissue impedance deviates from a predetermined acceptable range or threshold, triggering the system 6000 to issue a warning, a user override request, and / or assign a high display priority value to the user override request regarding the detected deviation.

[0110] In some embodiments, the trigger event may be the detection of a mismatch between a selected surgical instrument 21 and a surgical procedure performed by the surgical instrument 21. The mismatch may be detected, for example, by system 6000 and / or computer-implemented interactive surgical system 1. The type of surgical procedure and the inventory list of surgical instruments 21 used in the surgical procedure are entered through the user interface and / or may be detected, for example, through object recognition, object tracking, object labeling, and / or other suitable image processing techniques of live stream image frames, or through various suitable wired and / or wireless communication methods. The situation recognition module 6006 can compare the inventory list detected or entered by the user with a previously stored inventory list historically associated with the surgical procedure type detected or entered by the user. Upon detection of a mismatch, system 6000 assigns a high display priority value to warnings, user override requests, and / or confirmation requests related to the mismatch.

[0111] In at least one example, by detecting the selection of a circular stapler for use in a hysterectomy, the system 6000 assigns a high display priority value to a warning about the mismatch, a user override request, and / or a confirmation request. The system 6000 can then confirm the need for the circular stapler, remove the circular stapler from the current active list, or request staff to correct the mismatch in the procedure plan.

[0112] In some embodiments, the trigger event may be the detection of a non-conforming component in a surgical instrument assembly. Various surgical instruments 21 utilize replaceable components such as, for example, replaceable cartridges, reloads, end effectors, shafts, handles, motors, and / or batteries. Utilizing a non-conforming component may cause the surgical instrument 21 to malfunction, which may cause harm to the patient and / or interfere with the outcome of the surgical procedure. The system 6000 may assign a display priority value based on the detection of the non-conforming component.

[0113] The computer-implemented interactive surgical system 1 can detect non-conforming components through authenticity checks or integrity checks. Failure of authenticity and / or integrity verification may indicate a non-conforming component. In certain embodiments, various components are equipped with sensors that can detect proper connections indicating appropriate compatibility between connected components. In such embodiments, the sensor signal or its absence may indicate a non-conforming component.

[0114] In at least one example, if interchangeable components are installed within the surgical instrument 21, the surgical instrument 21 can query the interchangeable components for identification information that can be compared, for example, with recognized identification information stored in a database. The database may be maintained, for example, on the storage medium of the surgical instrument 21, hub 22, and / or remote server 13 of the cloud-based system 4. If authentication of the identification information fails, the system 6000 assigns a high display priority value to a warning, user override request, and / or confirmation request regarding the incompatible component. The computer-implemented interactive surgical system 1 may also block certain capabilities of the surgical instrument 21 or lock out the surgical instrument 21 in order to protect the patient and / or surgical outcome.

[0115] In some embodiments, the trigger event is the detection of a tissue condition, such as a biological abnormality, that could adversely affect the proper use of the surgical instrument 21 in a surgical procedure under standard settings. For example, an extremely high body mass index (BMI) may require adjustments to the various settings of the surgical instrument 21 in sleeve gastrectomy. The BMI level can be detected by the situation recognition module 6006, for example, from perioperative data.

[0116] Detecting a BMI level that deviates from an acceptable threshold may cause the system 6000 to assign a high display priority value to warnings, user override requests, and / or confirmation requests related to the BMI level. Furthermore, the system 6000 may assign an even higher display priority value to recommended surgical instrument settings, such as a lower firing rate for a surgical stapler used in sleeve gastrectomy. The system 6000 and / or the computer-implemented interactive surgical system 1 may be configured to automatically determine recommended surgical instrument settings based on perioperative data.

[0117] In various embodiments, determining the display arrangement of surgical data on the display 6005 (6013) includes modifying the characteristics of the visual representation of the surgical data. In some embodiments, the surgical data may be in the form of sensor readings that can be overlaid on a live stream of the surgical field on the display 6005. Sensor readings may be highlighted in a color that changes according to the importance of the sensor parameter readings to the surgical procedure. In some embodiments, sensor readings may be visually represented in a first color when the sensor readings are within a normal range of a given standard, but may be visually represented in a second color different from the first color when the sensor readings are outside a normal range.

[0118] For example, the sensor reading can be a temperature reading that can be visually represented in green when the temperature reading is below a predetermined temperature threshold. If the temperature reading exceeds the predetermined threshold, the temperature reading can be visually represented in yellow or red, for example, to indicate the importance of the current temperature for surgical procedures.

[0119] In some embodiments, changes in the characteristics of the visual representation of surgical data may be gradual transitions. For example, a temperature reading may gradually transition from yellow to red as the temperature rises to reflect the severity of the temperature change. In some embodiments, other characteristics of the visual representation may also be changed, such as size, shape, display time, display location, three-dimensional arrangement of the display (e.g., foreground, background), flashing of the display, highlighting, and / or font.

[0120] In various embodiments, determining the display arrangement of surgical data on display 6005 (6013) includes, for example, removing or modifying the characteristics of the visual representation of the surgical data to reflect reduced importance and / or inactive situations. In some embodiments, the surgical data includes the temperature of a surgical energy device used to seal tissue within the surgical field of a surgical procedure. In response to the activation of the surgical energy device, a visual representation of the temperature is overlaid on the live stream of the surgical field on display 6005. The visual representation means that the surgical energy device is "hot" to provide a warning for handling the surgical energy device carefully while it is in an active situation. In some embodiments, the visual representation may include characteristics indicating a high-priority situation to ensure that the attention of the clinician using the surgical energy device and / or other OR staff is drawn to it.

[0121] When clinicians use surgical energy devices, the visual representation of temperature may be assigned a lower priority status, even if the surgical energy device remains hot. This is to reduce clinician distraction and / or shift the clinician's attention to a different visual representation of higher-priority surgical data. For example, the visual representation of temperature may be changed to an intermediate color, reduced in size, and / or changed to a different shape.

[0122] When a surgical energy device becomes inactive, if the temperature is above a predetermined threshold, a high-priority status is reassigned to the temperature, changing its visual representation to provide a warning to draw attention to or emphasize that even inactive, the surgical energy device may still exceed a temperature threshold that could cause damage. In response to the temperature falling below the predetermined threshold, the visual representation of the temperature changes back to a lower-priority status. In some embodiments, the temperature of the surgical energy device can be monitored using one or more temperature sensors on or near the end effector of the surgical energy device. Sensor readings can be communicated to the system 6000 wirelessly or via wired communication.

[0123] In various embodiments, determining the display arrangement of surgical data (6013) includes transferring a visual representation of the surgical data between a first display and a second display. The transfer enables the system 6000 to present the surgical data to the appropriate user in a timely manner at an appropriate time and place. In some embodiments, the first display is a setup display, nurse display, or preparation display, and the second display is a surgical field or surgeon display, such as display 6005. In such embodiments, the transfer may be triggered by detection of the completion of the setup. In certain examples, user input may indicate the completion of the setup that triggers the transfer. The setup may include checking surgical devices against an inventory list to ensure the presence of the surgical devices necessary to perform the surgical procedure. The setup may further include testing the surgical devices to ensure the success of wireless communication operation, and / or any other appropriate tests.

[0124] In some embodiments, the control module 6001 is configured to assign a high display priority value to surgical data on the first display and a low display priority value to the same surgical data on the second display until a trigger event is detected. In response to detection, the control module 6001 is configured to assign a low display priority value to the surgical data on the first display and a high display priority value to the same surgical data on the second display. The priority switch transfers the surgical data to the second display. In some embodiments, the switch dims the visual representation of the surgical data on the first display and makes it visible on the second display. After a predetermined period of time, the visual representation of the surgical data can then be completely removed from the first display.

[0125] In various embodiments, the determined (6013) display configuration may require additional processing power, such as, for example, deploying a spectral view and / or keeping the surgical end effector in the surgical field, or overlaying surgical data onto the surgical end effector. Figure 17 is a flowchart illustrating the operation of exemplary method 60600 to respond to the need for additional processing speed during a surgical procedure performed by the computer-implemented interactive surgical system 1. When additional processing power is required (6061), the control module 6001 can utilize a field-programmable gate array (FPGA). For example, as shown in method 6060 of Figure 17, additional high-speed calculations of critical variables can be assigned to the FPGA in advanced visualization mode 6063. When advanced visualization mode 6063 is enabled, the FPGA is dynamically reused to maximize visualization (e.g., spectral) processing power. After the high-speed calculations are complete, the FPGA can return to normal operation in normal visualization mode 6062.

[0126] In some embodiments, the transition between normal visualization mode 6062 and advanced visualization mode 6063 may be triggered by a surgical task. The control module 6001 can detect the next or current surgical task based on contextual information generated by the situation awareness module 6006. The control module 6001 can refer to a database that may be stored in memory 6003 for visualization modes associated with surgical tasks. If a surgical task requires advanced visualization mode 6063, the control module 6001 repurposes the FPGA to assist with the high-speed computation associated with advanced visualization mode 6063. Once the surgical task is complete, the control module 6001 triggers a return to normal visualization mode 6062, effectively switching the FPGA to perform normal tasks.

[0127] In certain embodiments, detecting surgical data (6011) includes receiving two separate surgical data that are competing for the user's attention. For example, detecting surgical data (6011) may include receiving first surgical data and second surgical data, both of which are related to the current surgical task and / or associated with one or more active surgical devices. In such embodiments, method 6010 may include assigning display priority values ​​to the first surgical data and second surgical data based on a comparison of the severity of surgical success and / or failure that could result from ignoring them (6012). For example, if the first surgical data includes a higher severity than the second surgical data, method 6010 may assign a higher display priority value to the first surgical data than to the second surgical data (6012). In addition or alternatively, if a first failure associated with a first surgical data is more severe than a second failure associated with a second surgical data, method 6010 assigns a higher display priority value to the first surgical data than to the second surgical data (6012).

[0128] In some embodiments, display priority values ​​associated with various surgical data, as well as corresponding severity and / or failure severity, may be stored in a storage medium such as memory 6003 in any suitable format, e.g., a table or a database. The processor 85 of the control module 6001 may be configured to assign display priority values ​​(6012) based on such stored information.

[0129] In addition or alternatively, display priority values ​​can be assigned based on predetermined user preferences and / or user-specific surgical contexts (6012). In some embodiments, surgical data associated with an active surgical instrument 21 may be selectively displayed on a display associated with the clinician using the surgical instrument 21. Thus, method 6010 may include assigning different display priority values ​​to the same surgical data for different displays (6012).

[0130] In one example, surgical data associated with a first surgical device used by a clinician is simultaneously assigned a high display priority value to a first display selected by the clinician, or otherwise associated with the clinician, and a low display priority value to other displays not selected by the clinician or associated with the clinician (6012). In another example, first surgical data associated with a first surgical device used by a clinician is assigned a high display priority value to a first display selected by the clinician, or otherwise associated with the clinician, and second surgical data associated with a second surgical device not used by a clinician is assigned a low display priority value to the first display.

[0131] In various examples, the control module 6001 receives contextual information from the context awareness module 6006, which can be used in the aforementioned pairing of surgical data of a specific surgical device with a display associated with the clinician using the surgical device. The contextual information may be generated by the context awareness module 6006 based on perioperative data.

[0132] In some embodiments, a database or table can store the pairing information. In other examples, the clinician may wear a unique identifier that can be detected by the surgical device when the clinician holds the surgical device. Once positive identification is made, the control module 6001 can then assign a high display priority value to the surgical data associated with the surgical device for the display selected by the clinician or otherwise associated with it. In one example, the unique identifier may be an RFID in the clinician's glove, which is detected by a corresponding RFID scanner in the handle of the surgical device.

[0133] In certain cases, such as during colorectal procedures, the system 6000 is configured to automatically switch the display (e.g., display 6005) from showing a first live stream of a first surgical field to a second live stream of a second surgical field. The automatic switching can be triggered by the completion of a surgical task in the surgical procedure. In one example, a predetermined surgical cue indicating the completion of a surgical task can be used as a trigger for the automatic switching between live streams. The predetermined surgical cue may include, for example, detecting the completion of staple firing into tissue by the surgical instrument 21, detecting the completion of tissue sealing by the surgical instrument 21, and / or detecting the release of tissue from the jaws of the end effector of the surgical instrument 21, for example by opening the jaws.

[0134] A predetermined surgical queue may also include detecting the activation of the surgical instrument 21, followed by the deactivation of the surgical instrument 21, indicating the completion of a surgical task by the surgical instrument 21. In some embodiments, the control module 6001 utilizes readings from one or more sensors of the surgical instrument 21 and / or other components of the computer-implemented interactive surgical system 1 to detect the predetermined surgical queue. In some examples, the predetermined surgical queue is detected based on contextual information generated by the situation awareness module 6006.

[0135] In colorectal procedures, clinicians use circular and linear staplers to complete various tasks of the procedure. Colorectal procedures involve operating in two separate surgical fields: an internal field where diseased tissue is resected and an external field where the circular stapler is used. In some embodiments, a first live stream focuses on the internal compartment where tissue resection is taking place, and a second live stream focuses on the external compartment where the circular stapler is applied. In such embodiments, automatic switching can be triggered by the completion of tissue resection with the linear stapler, which can be detected, for example, by the stopping of the linear stapler and / or the removal of the linear stapler from the first surgical field. The control module 6001 may employ various object recognition, object tracking, and / or object labeling algorithms and / or other image processing techniques for the image frames of the live stream to detect, for example, the removal of the linear stapler from the surgical field.

[0136] Figure 18 is a flowchart illustrating the operation of an exemplary method 6070 for automatically switching live streams of surgical fields in a surgical procedure. In some embodiments, method 6070 may be performed, for example, by a computer-implemented interactive surgical system 1. Method 6070 includes presenting a first live stream of a first surgical field on a display (e.g., display 6005) (6071). When a predetermined surgical queue indicating the completion of a surgical task in the first surgical field is detected (6072), the system automatically switches from presenting the first live stream of the first surgical field on the display to presenting a second live stream of a second surgical field on the display (6073). In some examples, the second surgical field is associated with a second surgical task following a first surgical task in a surgical procedure.

[0137] During surgical procedures, various components of the computer-implemented interactive surgical system 1 may compete for available system resources, such as power, current, and / or processing resources. In addition, or alternatively, the operation of certain components of the computer-implemented interactive surgical system 1 may interfere with or adversely affect the operation of other components of the computer-implemented interactive surgical system 1. Various methods and systems are described herein to ensure that components function correctly by maintaining a balance of system resources and / or component operations.

[0138] Figure 19 is a flowchart illustrating the operation of an exemplary method 6050 for balancing system resources during a surgical procedure performed by a computer-implemented interactive surgical system 1. Method 6050 includes detecting failures of system resources that meet the competing needs of different components of the computer-implemented interactive surgical system 1 (6051). Method 6050 further includes displaying resource allocation controls for system resources in response to failure detection, for example, by overlaying resource allocation controls on a live stream of the surgical field of the surgical procedure (6052). In addition, Method 6050 may further include displaying recommended adjustments to the resource allocation controls.

[0139] In addition to the above, method 6050 includes adjusting the power consumption of one or more of the different components based on adjustments made by the user to resource allocation control (6053). Method 6050 may further include returning to the default resource allocation (6055) or removing the resource consumption limit when no more failures are detected. Method 6050 may further include, for example, displaying visual content that represents the effect of the adjustments to resource allocation by overlaying the visual content on a live stream of the surgical field on the display of the computer-implemented interactive surgical system 1 (6054), and / or displaying visual content that represents returning to the default mode (6056).

[0140] In some embodiments, fault detection (6051) includes reaching and / or exceeding predetermined thresholds, such as a power threshold, current threshold, processing threshold, and / or maximum utilization threshold. The predetermined thresholds may be selected to ensure that fault detection (6051) is achieved before power consumption exceeds the available power resources, in order to avoid malfunctions during surgical procedures. In some embodiments, the predetermined thresholds are stored in a storage medium, such as memory 6003, which is accessed by the processor 85 and compared with monitored values ​​(e.g., total consumption, consumption rate).

[0141] In some embodiments, a failure is detected (6051) when the control module 6001 detects that a competing task is being executed during a surgical procedure in which the total estimated resource consumption (e.g., power consumption) or resource consumption rate is above a predetermined threshold. In some embodiments, a failure is detected (6051) when the control module 6001 detects that multiple components of the computer-implemented interactive surgical system 1 are being used simultaneously in which the total estimated resource consumption (e.g., power consumption) or resource consumption rate is above a predetermined threshold. For example, a database stored in memory 6003 may include a list of resource consumption estimates associated with various components of the computer-implemented interactive surgical system 1 and / or various tasks performed by the computer-implemented interactive surgical system 1. The processor 85 can calculate resource consumption values ​​based on the information in the database and compare the calculated values ​​with predetermined thresholds to determine whether a failure has been detected (6051).

[0142] In some embodiments, the system resource is power, and the components of the computer-implemented interactive surgical system 1 that compete for power resources are system 6000, or any other visualization system of the computer-implemented interactive surgical system 1, and the generator 27. For example, during a surgical tissue sealing procedure, the computer-implemented interactive surgical system 1 may be configured to perform two tasks that collectively require power consumption to reach or exceed a predetermined threshold. The first task may be, for example, a visualization task that provides a spectral view of the surgical field, and the second task may be, for example, energizing a surgical energy device to seal tissue grasped by the surgical energy device in the surgical field. The generator module 27 may be configured to supply power to the surgical energy device to seal the tissue by applying therapeutic energy to it.

[0143] In such embodiments, a failure is detected by monitoring the power consumption of the system 6000 and the generator module 27 (6051). If the power consumption reaches and / or exceeds a predetermined threshold, the control module 6001 issues a user alert by causing an overlay 6052 of power allocation control onto the live stream of the surgical field on the display 6005. The control module 6001 can then adjust the power consumption according to the user's adjustment of the power allocation control.

[0144] In certain cases, the control module 6001 reduces the power requirements of one or more systems to perform user adjustments. For example, the control module 6001 may reduce the brightness of the display 6005 in response to a user input that selects to reduce the power allocation to system 6000 in order to prioritize maintaining the power allocation to generator module 27. In addition or alternatively, the control module 6001 may slow down, delay, or pause certain tasks performed by system 6000, such as secondary image processing tasks, in response to a user input that selects to reduce the power allocation to system 6000 in order to prioritize maintaining the power allocation to generator module 27.

[0145] In certain cases, user adjustments to the power allocation control may prioritize power allocation to the system 6000 over that to the generator module 27. This may occur when the user is in a critical process requiring optimal visibility, for example, when sealing a blood vessel, and when proper operation of the energy device can still be achieved at a lower power level, possibly by increasing the tissue sealing time. In such cases, the control module 6001 may cause the surgical energy device and / or the generator module 27 to adjust one or more of their settings to reduce power consumption in favor of the system 6000.

[0146] In some embodiments, the control module 6001 automatically intervenes to adjust power allocation without user input in response to fault detection. In such embodiments, the control module 6001 only alerts the user to changes caused by automatic changes to power consumption. For example, the control module 6001 may overlay alerts on the live stream on the display 6005 for changes in the brightness of the display 6005, and / or temporary suspension of visual content overlays, such as surgical data overlays, resulting from temporary interruptions in image processing that result in overlays. The overlays may be reintroduced upon completion of tissue sealing by the surgical energy device. Alternatively, the overlays may be displayed intermittently rather than continuously to reduce the power consumption of the system 6000 by prioritizing the generator module 27.

[0147] In some embodiments, user adjustments to power allocation control are implemented, for example, through one or more active individual current limiting circuits configured to prevent one or more systems from exceeding a maximum fuse limit threshold.

[0148] In some embodiments, the system resource is power, and the components of the computer-implemented interactive surgical system 1 that compete for power resources are system 6000, or any other visualization system of the computer-implemented interactive surgical system 1, and the smoke extractor module 26 (Figure 3). For example, during a surgical tissue sealing procedure, the computer-implemented interactive surgical system 1 may be configured to perform two tasks that collectively require power consumption to reach or exceed a predetermined threshold. The first task may be, for example, a visualization task that provides a spectral view of the surgical field, and the second task may be, for example, extracting smoke from the surgical field. The smoke is a byproduct of the tissue sealing process by an energy device.

[0149] In such embodiments, if a malfunction is detected (6051), the control module 6001 may then issue a user alert by, for example, causing an overlay 6052 of power allocation control on the live stream of the surgical field on the display 6005, as described above. The control module 6001 can then adjust power consumption according to the user adjustment of power allocation control. In a particular example, the control module 6001 may recommend adjusting the smoke exhaust module 26 to a lower setting by, for example, overlaying visual content representing the recommended adjustment on the live stream of the surgical field on the display 6005. In addition, the control module 6001 may also overlay visual content representing smoke exhaust deceleration. Presenting such visual content in the manner shown gives the user of the surgical energy device an opportunity to decelerate the sealing process by adjusting the surgical energy device to a lower setting that produces less smoke. For example, once the additional power requirements of the system 6000 are fulfilled due to the completion of image processing associated with the spectral view, the control module 6001 triggers an overlay of visual content representing an alert to inform the user that the smoke exhaust module 26 has returned to its original configuration.

[0150] In various examples, methods similar to method 6050 may be implemented to address other failures that could adversely affect surgical procedures performed using the computer-implemented interactive surgical system 1, such as overheating and / or noise. In such cases, failure detection may be achieved based on readings from one or more internal and / or external sensors of one or more components of the computer-implemented interactive surgical system 1. The sensor readings can then be compared to a predetermined threshold to detect the failure. For example, an overheating failure can be detected if one or more temperature sensor readings are above a predetermined temperature threshold. In response to the failure, the control module 6001 may overlay virtual controls on the live stream of the surgical field of the surgical procedure on the display 6005, thereby presenting the user with the opportunity to change the settings of one or more components of the computer-implemented interactive surgical system 1 to address overheating. A similar method can be used to address noise levels.

[0151] In various examples, the display arrangement according to method 6010 includes segmenting of the display 6005 to accommodate the visual representation of surgical data. The size, shape, display time, display location, three-dimensional display arrangement (e.g., foreground, background), display flashing, highlighting, and / or font of the segments displayed simultaneously may depend on several factors, including the nature, complexity, and / or severity of the surgical data. In some embodiments, pairing information for surgical data configured to be displayed simultaneously can be stored in a database or table on a storage medium such as memory 6003. The processor 85 of the control module 6001 may determine, based on the stored information, whether multiple surgical data should be displayed simultaneously.

[0152] In some embodiments, two different visual representations of surgical data are configured to be displayed simultaneously on the display 6005 in segmented mode, but only one of the visual representations is ready for display. In such embodiments, the unready visual representation can be represented as a blank area within its assigned segment. In addition, as described above, the control module 6001 may be configured to reuse the FPGA for additional processing speed to help prepare the unready visual representation. Alternatively, the unready visual representation may be displayed at a lower quality to ensure that the surgical data is displayed simultaneously.

[0153] In a particular example, visual representations of multiple surgical data are configured to be displayed simultaneously on display 6005, for example in segmented mode, but system 6000 lacks sufficient processing power to display all different surgical data simultaneously. In response to detecting this lack of processing power, system 6000 may, for example, prioritize the display of higher-priority surgical data over lower-priority surgical data based on the assigned display priority value of the surgical data.

[0154] In another example, a display problem may be that the display 6005 does not have sufficient display area to simultaneously display multiple visual representations of surgical data in segmented mode. In such a case, the display arrangement implemented by the control module 6001 may include a picture-in-picture type display arrangement in which the first visual representation is displayed inside the second visual representation. In other words, the first visual representation may appear in the foreground and may be smaller in size than the second visual representation which appears in the background. In addition, through any suitable user interface 6007, the clinician can switch between the two visual representations by selectively moving one of the visual representations to the foreground and the other to the background.

[0155] The control module 6001 may be configured to detect a lack of sufficient display area based on a predetermined display size of the display 6005 and a calculated display size of the visual representation of the surgical data. In some embodiments, a predetermined equation can be used for the calculation. In other examples, a lack of sufficient display is detected if the size of the visual representations is the same or similar, or if the number of visual representations of the surgical data is greater than or equal to a predetermined threshold.

[0156] In various examples, the display arrangement according to method 6010 includes transitions between display modes, such as static or passive display modes and dynamic or active display modes. In some embodiments, the control module 6001 is configured to transition the visual representation of surgical data from static mode to dynamic mode. The control module 6001 may be configured to perform transitions in response to predetermined triggers, such as changes in priority, severity, and / or risk associated with the surgical data. For example, surgical data initially assigned a low display priority value (6012) may be displayed or overlaid on a live stream of the surgical field in static display mode, which later transitions to active display mode as a result of the display priority value of the surgical data increasing to a higher display priority value.

[0157] In addition to the above, in some embodiments, the static mode includes displaying or overlaying a static visual representation of surgical data associated with the surgical instrument 21, for example, on the side or corner of the display 6005. In contrast, the active mode may include, for example, overlaying an active visual representation of surgical data on a portion of the surgical instrument 21 in a live stream of the surgical field, and / or moving the highlighted area in the static visual representation. In various embodiments, the static display mode differs from the active display mode in one or more of the following: size, shape, display time, display location, three-dimensional display arrangement (e.g., foreground, background), display blinking, highlighting, and / or font.

[0158] In some embodiments, the transition from static display mode to active display mode is based on the operation or activation of the surgical instrument 21, which signals a technology-sensitive process requiring real-time dynamic display. For example, the operation or activation of the surgical instrument 21 in a subsequent staple firing into tissue that requires a specific firing angle relative to a previous firing can trigger a transition to active display mode. Firstly, specific display elements, such as a visual representation of surgical data (e.g., various firing and / or tissue parameters), can be displayed or overlaid in static display mode. Then, in response to the operation or activation of the surgical instrument 21, in the subsequent firing, the control module 6001 transitions to a dynamic display mode in which, for example, the display elements are highlighted and / or moved. In various examples, the subsequent firing that triggers the transition involves a staple firing that also deploys tissue aids (e.g., tissue thickness compensators).

[0159] In some embodiments, the control module 6001 is configured to reduce the size of display elements in static display mode, reduce their emphasis, and / or erase display elements over time. Various operating parameters of the surgical instrument 21 can be first presented in dynamic display mode and then transitioned to static display mode as the significance level of such parameters changes. In certain examples, specific display elements are assigned to a predetermined location on the display 6005, for example, in static display mode and then modified in active display mode.

[0160] In some embodiments, a visual representation of surgical data, such as biomarkers, is presented in a static display mode, where, for example, solid colors are not highlighted, but the values ​​associated with the biomarkers remain within a predetermined range or below a predetermined threshold. However, if the values ​​move beyond a predetermined range or exceed a predetermined threshold, the visual representation of the surgical data may transition to a dynamic display mode by changing specific display elements of the visual representation, such as size, shape, display time, display location, three-dimensional display arrangement (e.g., foreground, background), blinking, highlighting, and / or font.

[0161] Figure 19A is a flowchart illustrating the operation of an exemplary method 6110 for transitioning between a static display mode and an active display mode based on surgical data. In some embodiments, method 6110 may be performed, for example, by a computer-implemented interactive surgical system 1. In the illustrated example, the surgical data includes tissue parameters. The tissue parameter is tissue impedance. Other tissue parameters such as tissue thickness, tissue pressure, tissue conductance, and / or tissue compression may be presented similarly.

[0162] In addition to the above, method 6110 includes detecting tissue between the jaws of the end effector of the surgical instrument 21 (6111). In a particular example, tissue detection 6111 may be achieved automatically, for example, through object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques for live stream image frames. Alternatively, the surgical instrument 21 may be configured to detect the presence of tissue between the jaws (61111) based on signal readings from one or more sensors in the jaws. For example, tissue may be detected when a non-therapeutic signal passing through the tissue results in an acceptable tissue impedance (6111).

[0163] In response to detecting tissue (6111), method 6110 presents the tissue parameters in a static display mode, for example, by displaying or overlaying a visual representation of the tissue parameters on a live stream of the surgical field (6112). However, if the tissue parameters reach or exceed a predetermined threshold or fall outside a predetermined range (6113), method 6110 further causes a transition 6115 to an active display mode for one or more display elements of the visual representation of the tissue parameters.

[0164] In some embodiments, the surgical instrument 21 is an energy device configured to seal tissue grasped by the end effector of the surgical instrument 21. At the start of treatment, upon detection of tissue (6111), the tissue impedance is presented in static display mode. The surgical instrument 21 can communicate surgical data indicating the tissue impedance to the control module 6001 via a wired or wireless interface, and display it, for example, on the display 6005 in static display mode. When energy is applied to the tissue, the tissue impedance changes. However, if the tissue impedance reaches or exceeds a predetermined threshold, or falls outside a predetermined range, this could be due to immersion of the end effector in fluid, an electrical short circuit, or simply low impedance tissue. In any case, a transition to active display mode 6115 is triggered to alert the clinician to investigate.

[0165] In various examples, the control module 6001 determines various surgical information associated with the surgical procedure, such as the steps of the surgical procedure, the surgical instruments 21 used in each step, and the various risks and / or techniques associated with each step. Such determinations may be based on contextual information generated, for example, by the situation awareness module 6006. The control module 6001 can then display or overlay the surgical information on the surgical field of the surgical procedure in a display configuration utilizing one or more of the methods described herein. For example, the current step, the surgical instruments 21 associated with the current step, the risks associated with the current step, and / or techniques associated with the current step may be presented in active display mode, while the previous and / or next step are presented in static display mode. When the next step becomes the current step, the display transitions to active display mode.

[0166] In addition to the above, the transition 6115 from static display mode to active display mode can be employed, for example, to reflect a new layout or to reflect changes to the treatment plan. In various examples, surgical information may be segmented for presentation by the control module 6001 into stages such as access, isolation and / or mobility, excision and / or repair and / or extension of related data to the surgeon.

[0167] In various examples, the transition of the visual representation of surgical data between static display mode and active display mode is based on changes in the use of surgical instruments 21 linked to or associated with the surgical data. The surgical data can initially be presented in static display mode. However, if a predetermined change is detected in the use of the surgical instruments 21, the transition of the visual representation of the surgical data to active display mode is affected.

[0168] Figure 19B is a flowchart illustrating the operation of an exemplary method 6120 for transitioning the visual representation of surgical data between a static display mode and an active display mode. The transition is based on or triggered by a change in the use of a surgical instrument 21 linked to or associated with the surgical data. In some embodiments, method 6120 may be performed, for example, by a computer-implemented interactive surgical system 1.

[0169] In the illustrated example, surgical instrument 21 is an ultrasonic surgical instrument configured to coagulate tissue grasped by its end effector during a surgical procedure. Surgical instrument 21 is used with a generator of a preset generator setting received by control module 6001 (6121) for display or overlay on the surgical field of the surgical procedure. Method 6120 further includes presenting the preset generator setting in static display mode (6122). However, if immersion of the end effector in blood is detected during the surgical procedure due to an attempt to coagulate a blood vessel partially immersed in blood (6123), for example, a new generator setting is presented in active display mode. The new generator setting may include an increase in the transducer power level in response to the immersion of the end effector in blood. Displaying or overlaying the new generator setting on the live stream of the surgical field alerts the user of surgical instrument 21 and gives the user an opportunity to adjust the position of the end effector if the increased power level is undesirable.

[0170] In some embodiments, detection of immersion of the end effector in blood is achieved by one or more sensors. In one example, a non-therapeutic current can be applied. If a short circuit is detected, the short circuit indicates immersion in blood. In response, surgical data indicating immersion is communicated to the control module 6001 wirelessly or via a wired interface.

[0171] In various examples, the display arrangement according to Method 6010 includes initially presenting a visual representation of surgical data in static display mode. Method 6010 then triggers changes in one or more display elements of the visual representation, such as values ​​associated with the surgical data, in response to changes in the status of the surgical instrument 21 associated with the surgical data. These changes include, for example, encountering a staple cartridge lockout, activating a high-energy device, or transitioning between an open and closed configuration of the end effector of the surgical instrument 21.

[0172] As described above, changes to one or more values ​​associated with surgical data can be performed in static display mode. Alternatively, in some embodiments, the change may be accompanied by a transition from static display mode to active display mode to provide additional alerts. Such embodiments include, for example, various adaptation techniques such as pausing to allow tissue creep and / or tissue compression, detecting unbalanced tissue in the jaws of the end effector of the surgical instrument 21, and / or detecting that the jaw clamp is inducing improper tissue tension.

[0173] In various examples, the display arrangement according to method 6010 includes a transition from a first dynamic display mode to a second dynamic display mode, the second dynamic display mode including or representing a higher priority, risk, and / or severity than the first dynamic display mode. In one example, blood pressure is tracked during a surgical procedure via a blood pressure monitoring device, which can communicate its readings to a control module 6001, for example, using a wireless or wired interface. The blood pressure data can then be presented visually in a first dynamic display mode, depending on its importance. However, if an increase in blood pressure data exceeding an acceptable limit is detected during the surgical procedure, a transition is made to raise the blood pressure data to a second dynamic display mode, for example, to ensure that an appropriate alert is delivered.

[0174] In various embodiments, one or more characteristics of the visual representation of surgical data, such as size, shape, display time, display location, three-dimensional display arrangement (e.g., foreground, background), display flashing, highlighting, and / or font, may be based on an assigned (6012) display priority value. In a particular example, the assigned (6012) display priority value may result in a display arrangement having display opposition. For example, determining the display arrangement based on an assigned display priority value may result in, for example, two or more visual representations of surgical data in the same location on display 6005.

[0175] Figure 20 is a flowchart illustrating the operation of an exemplary method 6010' for resolving display conflicts in display arrangement. Method 6010' is similar in many respects to Method 6010. Common details between the two methods are not repeated herein for the sake of brevity. In a particular example, as shown in Figure 20, a detected (6080) display conflict can be resolved by resolving the conflict (6081) by changing the display time of one or more of the conflicting visual representations. Alternatively, a clinician can recognize the conflict and be provided with choices (6082) to select (6082) between different surgical data on display 6005. Alternatively, the selection 6083 can be made automatically based on a predetermined preference of the clinician, which can be based on user input or contextual information generated by a situational awareness module, e.g., 6006.

[0176] In some embodiments, detecting a display conflict between first surgical data and second surgical data (6080) includes, for example, the processor 85 retrieving display priority information relating to the first surgical data and the second surgical data from, for example, memory 6003. The processor 85 can then compare the display priority information of the first surgical data and the second surgical data to determine whether a display conflict has been detected (6080).

[0177] In certain embodiments, the control module 6001 is configured to respond to detected (6080) display conflicts by, for example, simultaneously displaying competing visual representations of surgical data that are smaller in size than the default size. The clinician can, for example, select between the visual representations through the user interface 6007. In response, the control module 6001 removes the unselected visual representations and increases the size of the selected visual representations to the default size.

[0178] In certain embodiments, the detected (6080) display conflicts may be resolved by automatically prioritizing (6084) based on a resolution order determined based on the surgical data presenting the display conflicts. In some embodiments, the resolution order is determined based on the sequence of surgical steps associated with the surgical data, and / or the urgency of the risks and / or issues reported by the surgical data.

[0179] In a particular example, a display conflict is detected between the first surgical data and the second surgical data (6080), both presenting high-priority issues and / or risks. Furthermore, the second solution associated with the second surgical data cannot be performed until the first solution associated with the first surgical data has been performed. In such an example, the first visual representation of the first surgical data automatically takes precedence over the second visual representation of the second surgical data based on the resolution order (6084).

[0180] In a particular example, a display conflict may arise between first surgical data associated with a lockout preventing the operation of the surgical instrument 21 and second surgical data associated with the next best tissue thickness of the tissue being treated by the surgical instrument. In such an example, the tissue thickness issue is of higher priority but cannot be resolved while the surgical instrument 21 is in a lockout state; therefore, a predetermined resolution order may be adopted to resolve the conflict by prioritizing the lockout.

[0181] In certain examples, the resolution order may be stored on a storage medium (e.g., memory 6003) in the form of a database, a table, or any other suitable form. The stored information can enumerate various surgical data and their corresponding resolution orders. The processor 85 can refer to the stored information to identify the resolution order between conflicting surgical data and resolve the display conflict. In some embodiments, the resolution order is based on the order in which surgical tasks are started or completed based on the conflicting surgical data.

[0182] In some examples, the control module 6001 may receive first surgical data indicating that a detected staple cartridge (e.g., one loaded on a surgical instrument 21) has been previously fired. The controller of the surgical instrument 21 may query the staple cartridge, for example, by requesting firing information stored on the chip of the staple cartridge, and based on the retrieved firing information, may determine that the staple cartridge has been previously fired. The first surgical data, including the firing information, may be communicated to the control module 6001 wirelessly or via radio communication. Furthermore, the control module 6001 may receive second surgical data associated with the closure of the end effector of the surgical instrument 21 onto the tissue being stapled in a surgical procedure involving the surgical instrument 21 loaded with a previously fired staple cartridge. For example, the second surgical data may relate to the tissue thickness and / or tissue location between the jaws of the end effector.

[0183] In addition to the above, the control module 6001 detects a display conflict (6080) because both the first surgical data, i.e., a previously fired staple cartridge, and the second surgical data, i.e., the closure of the end effector onto the tissue, involve high-priority situations. To determine the display arrangement of the visual representations of the first and second surgical data on the display 6005, the processor 85, for example, checks the resolution sequence information stored on the storage medium (e.g., memory 6003) in the form of a database, a table, or any other suitable form. In this example, the first problem presented by the first surgical data, i.e., a previously fired staple cartridge, must be resolved before the second problem presented by the second surgical data, i.e., the closure of the end effector onto the tissue. This is because resolving the closure of the end effector onto the tissue is not important if the tissue cannot be treated using the previously fired staple cartridge.

[0184] Once the display conflict is resolved, method 6010' proceeds to display the visual representations of the first surgical data and the second surgical data according to the display arrangement selected based on the resolution order (6014'). For example, the first visual representation of the first surgical data may be displayed before the second visual representation of the second surgical data. Other suitable display arrangements may be employed, as described elsewhere in this disclosure.

[0185] In various embodiments, the surgical procedure involves stapling tissue using a surgical instrument 21, such as a surgical stapler. The surgical procedure typically includes positioning the end effector of the surgical instrument 21 within the surgical field and operating the end effector to grasp the tissue between its jaws. The jaws place the grasped tissue under compression. Because the tissue contains water, the grasped tissue gradually changes in response to compression by the jaws of the end effector in a process known as tissue creep until the tissue reaches a steady state. Furthermore, the gap between the jaws and the thickness of the tissue may also change until the tissue reaches a steady state. Tissue flow or movement may also occur until the tissue reaches a steady state. In some embodiments, to ensure successful stapling, the tissue is allowed a waiting period to achieve a steady state. Parameters associated with the aforementioned tissue changes, such as the waiting time parameter, tissue thickness parameter, and / or instrument gap parameter, are important for appropriately evaluating when the tissue reaches a steady state.

[0186] Figure 21 is a flowchart illustrating the operation of an exemplary method 6090 for addressing tissue changes (e.g., tissue creep, tissue flow, tissue compression) in a surgical procedure employing a surgical instrument 21. In some embodiments, method 6090 includes detecting tissue between the jaws of the end effector of the surgical instrument 21 (6091). In certain examples, tissue detection 6091 may be achieved automatically and visually, for example, through object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques for live stream image frames. Alternatively, the surgical instrument 21 may be configured to detect the presence of tissue between the jaws (6091) based on signal readings from one or more sensors within the jaws. For example, tissue may be detected when a non-therapeutic signal passing through the tissue results in an acceptable tissue impedance (6091).

[0187] Depending on the tissue detection 6091, method 6090 may display or overlay (6092) at least one parameter of tissue change (e.g., tissue creep, tissue flow, tissue compression), and / or a parameter of the gap distance between the jaws of the end effector, and / or a waiting time, on a live stream of the surgical field. In certain embodiments, method 6090 further includes alerting the user of the surgical instrument 21 when a steady state is reached to initiate tissue treatment (6094). In certain examples, a steady state is detected based on one or more of the tissue change parameters and / or one or more of the surgical instrument parameters (6093). For example, a steady state can be detected when one or more of the tissue flow, tissue creep, tissue thickness, tissue compression, gap distance between the jaws of the end effector, and / or waiting time are above a predetermined threshold (6093). Alternatively, a steady state can be detected when the rate of change of one or more of the following is below a predetermined threshold: tissue flow, tissue creep, tissue thickness, tissue compression, gap distance between jaws of end effectors, and / or latency (6093). In addition or alternatively, a steady state can be automatically detected visually based on object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques capable of monitoring changes in tissue, for example (6093).

[0188] In some embodiments, method 6090 further includes visually and automatically monitoring tissue changes during the application of treatment with the surgical instrument 21 by, for example, utilizing one or more suitable object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques for live stream image frames. In certain examples, the treatment may be, for example, firing staples into the grasped tissue. If the tissue change reaches an excessive level during firing (6095), method 6090 may further include displaying or overlaying an alert to the clinician (6096). In certain examples, method 6090 includes displaying or overlaying a visual representation of the location and / or size of the excessive tissue change (6097), as shown, for example, in Figure 26. In some embodiments, the tissue change is automatically and visually monitored by, for example, tracking the size, location, color, and / or movement of one or more tissue targets within the grasped tissue.

[0189] Method 6090 may also include displaying or overlaying a recommended solution (6098) that involves adjusting one or more parameters of the surgical instrument 21, such as one or more closure parameters (e.g., jaw clamping, jaw pressure, distal tip load) and / or firing parameters (e.g., firing rate, I-beam rate). In a particular example, the recommended solution may be an additional waiting time. In a particular example, the surgical instrument 21 is an ultrasound instrument, and the recommended solution is to reduce the distal tip load of the end effector. In another example, the surgical instrument 21 is a surgical stapler, and the recommended solution is to increase the distal tip load of the end effector.

[0190] In various cases, tissue changes (e.g., tissue flow) are affected, at least partially, by the tension experienced by the tissue gripped between the jaws. In certain cases, tissue tension is due to movements such as rotation of the end effector from a neutral position while gripping the tissue. In such cases, the overlaid (6098) solution may be a preferred form of adjustment for the rotational position of the end effector. Excessive tissue tension can be automatically observed, for example, by utilizing one or more suitable object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques for live stream image frames.

[0191] In some embodiments, the position and / or orientation of the end effector can be determined using one or more sensors, including an accelerometer, a gyroscope, a relative position sensor, and / or a three-dimensional magnetic sensor. In some embodiments, the sensors can generate position information that characterizes one or more position changes. The position information can be transmitted to the control module 6001 via a wired or wireless interface.

[0192] In various embodiments, the accelerometer may be a single-axis, two-axis, or three-axis accelerometer. The accelerometer may be employed to measure appropriate acceleration that is not necessarily coordinate acceleration (rate of change of velocity). Instead, the accelerometer can observe acceleration associated with the weight phenomenon experienced by a test mass at rest within the accelerometer's reference frame. In addition or alternatively, the position and / or orientation of the end effector can be observed automatically, for example, by utilizing one or more suitable object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques of the live stream image frame.

[0193] In response to the detection of tissue tension in the tissue grasped by the jaws of the end effector, the control module 6001 may display or overlay a visual representation of the tissue tension, its magnitude, and / or the rotational orientation involved in the tissue tension on the live stream of the surgical field. In some embodiments, as shown in Figures 22A-22C, the visual representations of tissue tension 6100, 6101, 6102 may provide, for example, positional information of the end effector in three-dimensional space. In some embodiments, the positional information of the end effector is represented by a first axis (e.g., x-axis) extending longitudinally through the end effector to the center, a second axis (e.g., y-axis) perpendicular to the first axis and extending in a first plane having the first axis, and a third axis (z-axis) perpendicular to the first axis and extending in a second plane having the first axis, the first plane intersecting the second plane at the first axis.

[0194] As shown in Figure 22A, while the end effector is neutral with respect to the coordinate axes, each of the coordinate axes may be presented in a first form (e.g., color, shape, size). In response to detecting an excessive deviation from the neutral state for one or more coordinate axes, the control module 6001 changes one or more coordinate axes to a second form different from the first form. In other examples, the excessive deviation from the neutral state may be a first deviation and may be based on a first predetermined threshold or range, while the second deviation may be more excessive than the first deviation and may be based on a second predetermined threshold or range different from the first predetermined threshold or range. In such cases, the neutral state may be presented in a first form, the first excessive deviation may be presented in a second form, and the second excessive deviation may be presented in a third form different from the first and second forms. In certain embodiments, the first form includes green, the second form includes yellow, and the third form includes red.

[0195] In the illustrated example, a first excessive deviation from the neutral state is detected around the y-axis. In response, the control module 6001 switches the y-axis from the first to the second state, while the x-axis and z-axis remain in the first state, as shown in Figure 22B. In the illustrated example, the first excessive deviation is above a first predetermined threshold. Next, as shown in Figure 22C, a second excessive deviation above a second predetermined threshold is detected around the x-axis, with the first excessive deviation around the x-axis improved. In response, the control module 6001 returns the y-axis to the first state and changes the x-axis to the third state.

[0196] In various examples, different deviations from the neutral state (e.g., first and second excessive deviations) can include different severities and can be presented in different forms indicating severity. For example, a first excessive deviation may be shown in yellow, and a second excessive deviation, which is more severe than the first, may be shown in red. In some embodiments, deviations from the neutral state are determined based on a range of rotation angles around one or more of the coordinate axes. For example, the neutral state with respect to the first axis is detected when the rotation angle of the end effector with respect to the first axis is in the range of approximately ±A°, a first excessive deviation is detected when the rotation angle of the end effector with respect to the first axis is in the range of approximately ±B°, and a second excessive deviation is detected when the rotation angle of the end effector with respect to the first axis is in the range of approximately ±C°. In the illustrated example, A, B, and C are integers, where A is less than B and B is less than C.

[0197] Referring to Figures 23A and 23B, in some embodiments, the visual representations of tissue tension 6104, 6105 may further include tissue tension measurements 6104a, 6105a associated with each of the coordinate axes. The control module 6001 may change the form (e.g., color, size, and / or shape) of the tissue tension measurements in response to an excessive deviation of tissue tension (e.g., from 2.0 lbs to 7.0 lbs).

[0198] In some embodiments, the control module 6001 may further display or overlay recommendations on a live stream of the surgical field to address excessive tissue tension. In some examples, as shown in Figure 24, the recommendation includes a visual representation 2106 showing the surgical instrument 21 along with an arrow 6107 representing the recommended rotation for transitioning the end effector of the surgical instrument 21 to a neutral position.

[0199] Figures 24–30 show various display configurations determined based on (6013) detected (6011) surgical data according to method 6010 and / or any other suitable method of this disclosure. The display configurations shown in Figures 24–30 are represented in relation to surgical instruments 21 configured to staple and cut tissue. However, in other embodiments, one or more of the display configurations shown in Figures 24–30 can be similarly used with other surgical instruments in other types of surgical procedures.

[0200] Some of the display arrangements described herein involve overlaying various visual representations of surgical data onto a live stream of the surgical field displayed on a display, for example, display 6005. As used herein, the term overlay includes translucent overlays, partial overlays, and / or moving overlays. Furthermore, overlays may be placed on, or at least partially on or near, objects in the surgical field, such as end effectors and / or important surgical structures. A particular display arrangement may include changes in one or more display elements of the overlay, including changes in color, size, shape, display time, display location, display frequency, highlighting, or combinations thereof, based on a change in display priority value.

[0201] Figure 25 shows a display configuration 6117 that includes a mixed reality view presented, for example, on a display 6005 by a control module 6001. The display 6005 shows a live stream of the surgical field during a surgical procedure in which the surgical instrument 21 is used to staple and cut tissue T grasped by the end effector 6119 of the surgical instrument 21. In the illustrated example, the display configuration 6117 overlays a cutting progress line 6118 or a staple firing progress line on the channel of the end effector 6119. Furthermore, the display configuration 6117 overlays the distance D traveled by the firing member or cutting member on the channel of the end effector 6119 to help the clinician follow the firing progress of the surgical instrument 21.

[0202] In some embodiments, the control module 6001 detects, for example, a change in one or more parameters of the tissue grasped by the end effector 6119 and / or a parameter of the surgical instrument 21 that exceeds a predetermined threshold or range. In at least one embodiment, the parameter change is a change in firing rate below a predetermined threshold. For example, the control module 6001 may receive surgical data indicating the parameter change through a wired or wireless communication interface with the surgical instrument 21 and / or the surgical hub 6 (Figure 1). In response to the detection of a parameter change, the control module 6001 may trigger a change in the cutting line 6118 or staple firing line on the channel of the end effector 6119, including changes in color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof.

[0203] In addition or alternatively, in response to the detection of a parameter change, the control module 6001 may change at least one color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof in the overlay of the virtual channel overlaid on the end effector 6119, according to the magnitude of the change, according to the parameter value, or according to the risk level associated with the parameter change.

[0204] Figure 26 shows, for example, a display arrangement 6114 presented by a control module 6001 on a display 6005 according to the method of the present disclosure. The display 6005 shows a live stream of the surgical field during a surgical procedure in which a surgical instrument 21 is used to staple and cut tissue T grasped by the end effector 6119 of the surgical instrument 21. In the illustrated example, display arrangement 6117 overlays tissue markers 6116 indicating tissue flow onto tissue T. Excessive tissue flow can be detected, for example, as described in relation to method 6090 of Figure 21. In the illustrated example, display arrangement 6114 combines an overlay of a cutting line 6118 and tissue markers 6116. Other display arrangements may include only tissue markers 6116.

[0205] Figures 27A–27C show a display arrangement 6130 that provides a visual representation 6131 of surgical data according to at least one aspect of the present disclosure. In some embodiments, the display arrangement 6130 is presented by a control module 6001 on a display 6005, for example, according to the method of the present disclosure. In the illustrated example, the display arrangement 6130 presents a visual representation 6132 in the form of a translucent overlay 6133 that shows tissue flow during a firing sequence of a surgical instrument 21. During the firing sequence, the surgical instrument 21 is configured to deploy staples into tissue T grasped by the end effector of the surgical instrument 21 and simultaneously cut the tissue T. In the illustrated example, the display arrangement 6130 is presented in dynamic display mode, showing changes in the display elements 6132 (Figure 27A), 6133' (Figure 27B), and 6133'' (Figure 27C) of the visual representation 6131.

[0206] Display elements may track tissue flow across the width of the end effector. Different locations may be presented in different forms (e.g., color, shape, and / or size), and different forms may represent different levels of tissue flow at different locations. In the illustrated example, display element 6132 represents an acceptable tissue flow state, and display element 6132' represents a low-risk tissue flow state. Conversely, display element 6132''' represents a high-risk tissue flow state.

[0207] Referring to Figures 28 and 29, in some embodiments, the display arrangement 6140 is presented by the control module 6001, for example, on a display 6005, according to the method of this disclosure. The display 6005 shows a live stream of the surgical field during a surgical procedure in which the surgical instrument 21 is used to staple and cut tissue T grasped by the end effector 6141 of the surgical instrument 21. In the illustrated example, the display arrangement 6140 overlays a performance parameter plot 6142 (Figure 29) along with a history trace as a function of the firing member, cutting member, and / or knife position. The plot 6142 is overlaid, for example, adjacent to the end effector 6141.

[0208] Plot 6142 presents the risk severity associated with one or more parameters 6146 monitored during the firing sequence of the surgical instrument 21, such as anvil gap, tissue load, firing velocity, and / or motor speed. Furthermore, plot 6142 further provides multiple thresholds, for example, three thresholds 6143, 6144, and 6145, each representing a severity level (e.g., low, medium, high), to provide clinicians with a visual indicator of the risk severity associated with the measured parameter 6146.

[0209] In addition or alternatively, the display configuration 6140 may be configured to present surgical data associated with tissue parameters (e.g., tissue pressure, tissue compression, tissue flow, tissue thickness) of tissue T using a color plot 6147. Tissue parameter values ​​may be represented by different colors (e.g., green, yellow, red, or light, medium, or dark) according to the relationship of the values ​​to one or more predetermined thresholds. In the illustrated example, green represents tissue portions with acceptable values, yellow represents tissue portions with low-risk values, and red represents tissue portions with high-risk values. The color plot 6147 provides a convenient and rapid risk assessment tool to assist clinicians in deciding, for example, whether to initiate and / or continue a firing sequence.

[0210] In various embodiments, tissue parameter values ​​are measured, for example, by sensors distributed at multiple locations across the width and length of the end effector 6141. The tissue parameter values ​​are then represented, for example, by colored areas on a color plot 6147 corresponding to the sensor locations on the end effector 6141 (e.g., green, yellow, red, or light shades, medium shades, dark shades).

[0211] Figure 30 shows a display arrangement 6150 that provides a visual representation of surgical data according to at least one aspect of the present disclosure. In some embodiments, the display arrangement 6150 is presented by a control module 6001 on a display 6005, for example, according to the method of the present disclosure. In some embodiments, the display arrangement 6150 is overlaid on a live stream of the surgical field of a surgical procedure in which tissue is stapled and cut using surgical instruments 21.

[0212] In some embodiments, the display arrangement 6150 includes a simulated cross-sectional overlay 6152 of the end effector 6153 of the surgical instrument 21, which, for example, shows and matches the position and movement of one or more end effector components in real time. Improved visualization can help clinicians better understand the current situation and risk-based feedback from the surgical instrument 21 (e.g., clamp load too high, firing force too high, required waiting time, etc.).

[0213] In the illustrated example, the simulated overlay 6152 shows staples 6156, a staple driver 6154, and a launcher (e.g., thread 6155) configured to motivate the staple driver 6154 to deploy the staples 6156 within the tissue. The position of the launcher within the simulated overlay 6152 reflects the position of the launcher within the end effector 6153, showing the progress of the launch sequence in real time. Furthermore, in the illustrated example, the simulated overlay 6152 shows simulated tissue (ST), which may be presented to reflect tissue flow in the region where tissue flow is detected. While the illustrated example shows only one column of staples 6156, other examples may show multiple columns.

[0214] In some embodiments, the firing sequence is represented by an overlay 6152 simulated in dynamic display mode. Furthermore, staple formation can, in some cases, be predicted based on one or more determined parameters such as tissue type, patient parameters, tissue flow, closure force, tissue creep stability, and anvil gap. For example, the control module 6001 can predict staple formation by employing a predetermined equation, database, and / or table.

[0215] In the illustrated example, the display arrangement 6150 further includes a stapling overlay 6157. The control module 6001 may be configured, for example, to predict stapling and update the stapling overlay 6157 in real time.

[0216] Figure 31 is a flowchart illustrating the operation of an exemplary method 6160 for risk-based operation of display placement during a surgical procedure, according to at least one aspect of the present disclosure. In some embodiments, method 6120 may be performed by, for example, a computer-implemented interactive surgical system 1. In some embodiments, method 6160 is performed by a surgical system comprising surgical instruments 21 configured to staple and cut tissue in the surgical field of a surgical procedure. The surgical system further comprises a control module 6001, an imaging device 6004, and a display 6005 configured to show a live stream of the surgical field. The live stream is captured, for example, by the imaging device 6004.

[0217] In some embodiments, method 6160 includes detecting a surgical risk (6161), assigning a severity level to the surgical risk (6162), and determining a display placement based on the severity level (6163), the display placement including overlaying an alert feature on a live stream. In some embodiments, method 6160 further includes presenting a visual representation of the surgical risk according to the display placement (6164).

[0218] In some embodiments, surgical risks are detected by the control module 6001 (6161). Surgical risks can be detected based on a single surgical data received from one or more sources, such as components of the computer-implemented interactive surgical system 1 via one or more wireless and / or wired communication interfaces (6161). In at least one example, the surgical data may include data received from one or more surgical instruments 21. In another example, the surgical data may include contextual information confirmed by the situation awareness module 6006.

[0219] In specific examples, surgical data may include control data, biomarker measurements, and / or other operational indicators of operation and / or results associated with the surgical instrument 21. In specific examples, surgical data may be any data indicating a higher tendency for malformed staples and / or poorly sealed tissue. In specific examples, surgical data may be associated with tissue flow, clamping force, and firing force, among other tissue and / or instrument parameters, which can be monitored and displayed to the clinician in real time in multiple ways to allow adjustment of the firing sequence or to alert the surgeon of potentially malformed staple areas.

[0220] In a particular example, the processor 85 employs a predetermined equation and / or formula when determining the severity level of surgical risk. Various relevant factors may be considered, and different weights may be assigned when calculating the severity level. In addition or alternatively, one or more databases or tables listing surgical data and corresponding severity levels may be available to the processor 85 when assigning (6162) the severity levels. In various embodiments, the assigned (6162) severity levels include, for example, low severity, moderate severity, or high severity.

[0221] Figure 32 shows an embodiment of the display arrangement 6170 according to at least one aspect of the present disclosure. In some embodiments, the display arrangement 6170 is determined, for example, based on the severity level of the surgical risk detected in method 6160 (6161). In the illustrated example, the display arrangement 6170 includes the control module 6001 overlaying an alert feature 6171 in response to the detection of a surgical risk 6161. The alert feature 6171 is overlaid on a live stream of the surgical field 6179 during the surgical procedure. In the illustrated example, the live stream of the surgical field 6179 shows an end effector 6172 of a surgical instrument 21 configured to operate a surgical structure 6178.

[0222] In the illustrated example, the alert feature 6171 is overlaid on the live stream in a corner region, away from the end effector 6172 and / or any significant surgical structures, so as not to obstruct the clinician's view of the surgical field. In other examples, the alert feature 6171 may be moved to a central region of the live stream, closer to the end effector 6172 and / or any significant surgical structures, or initially overlaid on them, for example, to indicate a higher severity of surgical risk.

[0223] In addition to the above, display configuration 6170 includes modifying alert feature 6171 in response to user responses. In the illustrated example, the modification to alert feature 6171 includes replacing alert feature 6171 with information 6173 associated with surgical risk. Information 6173 may include details about surgical risk and / or recommended solutions.

[0224] In the illustrated example, the user response is a transition of the end effector 6172 between an open configuration and a closed configuration. In other embodiments, the user response may include any other appropriate gesture or movement by the end effector 6172. In yet another embodiment, the user response may include, for example, a hand gesture or movement and / or an eye gesture or movement.

[0225] In other examples, user responses may be composite or multifactor responses to ensure that accidental actions by the user are not interpreted by the control module 6001 as user responses intended to manipulate the alert feature 6171. In some porting, a user response recognizable by the control module 6001 may include two components, such as a gesture or movement of the end effector followed by, for example, the user's eye movements or the user's hand movements.

[0226] In some embodiments, a display configuration 6170', which is similar in many respects to display configuration 6170 as shown in Figure 33, includes a different user response. In the illustrated example, the user response includes hovering the end effector 6172 over the alert feature 6171.

[0227] In some embodiments, user responses are automatically detected, for example, through object recognition, object tracking, object labeling, and / or other suitable image processing techniques for live stream image frames, or through various suitable wired and / or wireless communication methods. In addition or alternatively, user responses can be automatically detected by receiving information indicating user responses via suitable wired and / or wireless communication methods. For example, a camera may monitor the user's body movements or body gestures, such as waving, squinting, or double blinking. In another example, a clinician's glove can be tracked via one or more suitable sensors positioned on the glove. Sensor readings indicating a predetermined hand movement and a predetermined user response can be communicated to the control module 6001.

[0228] In some embodiments, the display arrangement 6170 includes modifying the alert feature 6171 based on changes in the severity of the surgical risk. The modification can be carried out, for example, in a dynamic display mode. In some examples, the modification to the alert feature 6171 includes at least one change to the color, size, shape, display time, display location, display frequency, highlighting, or a combination thereof, according to the severity level of the surgical risk. In some embodiments, the alert feature 6171 is, for example, in the form of an alert icon that changes color based on the severity level of the surgical risk.

[0229] In some embodiments, display configuration 6170'', which is similar in many respects to display configuration 6170 as shown in Figure 34, includes positioning an alert feature 6171 in a location that obstructs the clinician's view of a critical surgical structure 6178 to indicate a high severity surgical risk, for example. Nevertheless, display configuration 6170'' allows the user to move the alert feature 6171 away from the critical surgical structure 6178 by hovering the end effector 6172 over the alert feature 6171. In response to detecting that the end effector 6172 and the alert feature 6171 occupy the same location, the control module 6001 moves the alert feature to a different location on the live stream of the surgical field 6179, for example.

[0230] In other examples, a predetermined user response, such as opening and closing the jaws of the end effector 6172 once or twice, simulates grasping the alert feature 6171. Furthermore, the end effector 6172 may move, for example, to the corner of the display 6005, moving the grasped alert feature 6171 with it. Pausing at the new location may indicate dropping the alert feature at the new location. Other appropriate gestures and / or movements may be employed to indicate a user response that moves the alert feature 6171 away from the critical surgical structure 6178. In some embodiments, for example, in dynamic mode, the control module 6001 may automatically move the alert feature 6171 away from the end effector 6172 and / or critical surgical structure 6178 after an initial deployment determined to be, for example, below the end effector 6172 and / or critical surgical structure 6178.

[0231] In various embodiments, gestures and / or movements by the end effector 6172 may be automatically observed, for example, by utilizing one or more suitable object recognition, object tracking, and / or object labeling algorithms, and / or other image processing techniques for live stream image frames of the surgical field 6179. In various examples, the end effector 6172 is visually recognized based on characteristic reflectivity, color, and / or shape. In addition or alternatively, gestures and / or movements by the end effector 6172 may be detected through sensor readings of sensors within the surgical instrument 21.

[0232] In some embodiments, a change in the alert feature 6171 in response to a user response includes the movement of the alert feature 6171 away from the end effector 6172 and / or critical surgical structure 6178. In some examples, the control module 6001 is configured to track the position of the end effector 6172 and / or critical surgical structure 6178 relative to the position of the alert feature 6171 on the display 6005, for example. Furthermore, the control module 6001 is configured to automatically change the position of the alert feature 6171 based on the position of at least one of the end effector 6172 and / or critical surgical structure 6178, for example, to facilitate clear visibility of the end effector 6172 and / or critical surgical structure 6178.

[0233] In some embodiments, the control module 6001 is configured to correlate, for example, an alert feature 6171 to the source of the risk represented by the alert feature 6171. The correlation provides the clinician with an indication of the nature of the risk without the need to extend the alert feature 6171 to visualize the details of the risk. The correlation can be achieved through common display characteristics, such as common color highlighting and / or a common flashing frequency. For example, if the risk is associated with a surgical instrument 21 that includes an end effector 6172 in the surgical field, both the alert feature 6171 and the end effector 6172 may be highlighted with a common color, for example. In addition or alternatively, the correlation can be achieved by causing the surgical instrument 21 to provide audible and / or haptic feedback that matches the presence of the alert feature 6171 on the display 6005. In addition or alternatively, correlation can be achieved by overlaying one or more color-coded bubbles and / or arrows, separate from the alert feature 6171 that points to the end effector 6172, indicating that the risk represented by the alert feature 6171 is associated with the surgical instrument 21.

[0234] In some embodiments, the display arrangement associated with a particular surgical instrument task or surgical procedure may be changed in response to the detected completion of the surgical instrument task or surgical procedure. For example, a surgical procedure such as a surgical sleeve procedure involves a predetermined number of firings of a surgical instrument 21 configured to staple and cut tissue. Each firing in the firing sequence deploys staples from a staple cartridge into the tissue. The staple cartridge is then replaced with a new staple cartridge for the next firing in the firing sequence. The control module 6001 may be configured to detect the number of firings by the surgical instrument 21 and continue to overlay the surgical data associated with the firings of the surgical instrument 21 until a predetermined number of firings is reached. In response to the detection of completion of firings, the control module 6001 collapses or removes the overlay of the surgical data associated with the firings of the surgical instrument 21 from the display 6005.

[0235] In some embodiments, the detection of the completion of a surgical instrument task or surgical process can be achieved visually and automatically, for example, through object recognition, object tracking, object labeling, and / or other suitable image processing techniques for live stream image frames, or, for example, through input from the surgical instrument 21 and / or surgical hub 6 via various suitable wired and / or wireless communication methods.

[0236] In various examples, one or more functions of the aforementioned method are performed by one or more components of the computer-implemented interactive surgical system 1, such as one or more components of the surgical visualization system 6000. In certain examples, the components that perform one or more functions of the aforementioned method communicate via wireless and / or wired communication interfaces. In various examples, the memory of the computer-implemented interactive surgical system 1, for example, memory 6003, stores program instructions that, when executed by a processor (e.g., processor 85), cause the processor to perform one or more functions of the aforementioned method. Although the aforementioned functions are described separately, in some embodiments, some functions of the aforementioned method can be combined in any suitable form to produce different methods, for example, resulting in different program instructions for execution by one or more components of the computer-implemented interactive surgical system 1.

[0237] In various examples, to perform tracking, an algorithm analyzes a sequence of video frames and outputs the target's movement between frames, according to one or more aspects of this disclosure. Illustrative algorithms include target representation and localization algorithms, as well as filtering and data association algorithms. Target representation and localization algorithms include, for example, kernel-based tracking and / or contour tracking. Filtering and data association algorithms include, for example, Kalman filters and particle filters.

[0238] Various additional aspects of the subject matter described herein are illustrated in the following numbered examples. Example 1. A surgical system is intended for use in surgical procedures. The surgical system comprises a surgical instrument configured to treat tissue in a surgical procedure, an imaging device, a display configured to show a live stream of the surgical field of the surgical procedure, the display of which the live stream is captured by the imaging device, and a control module. The control module is configured to detect surgical risks, assign display priorities to surgical risks, and determine the display placement of surgical risks based on the display priorities, wherein the display placement includes overlaying alert features on the live stream.

[0239] Example 2. The surgical system described in Example 1, wherein the alert feature is an alert icon.

[0240] Example 3. The surgical system according to Example 1 or 2, wherein the display arrangement includes overlaying alert features on the corners of the live stream.

[0241] Example 4. A surgical system according to any one of Examples 1 to 3, wherein the display arrangement includes changing the alert features based on the severity level of the surgical risk.

[0242] Example 5. A surgical system according to any one of Examples 1 to 4, wherein the surgical risk is related to the level of tissue bleeding.

[0243] Example 6. A surgical system according to any one of Examples 1 to 4, wherein the surgical risk is associated with tissue parameters.

[0244] Example 7. A surgical system according to any one of Examples 1 to 4, wherein the surgical risk is associated with the parameters of the surgical instrument.

[0245] Example 8. A surgical system according to any one of Examples 1-4, wherein the surgical risk is associated with tissue movement.

[0246] Example 9. A surgical system according to any one of Examples 1 to 3, further comprising the display arrangement causing a change in the alert feature in response to a user response to the alert feature.

[0247] Example 10. A surgical system according to any one of Examples 9, wherein the surgical instrument comprises an end effector, and a user response triggers a transition of the end effector in the surgical field between an open configuration and a closed configuration.

[0248] Example 11. A surgical system according to any one of Examples 9, wherein the modification of the alert feature includes replacing the alert feature with information associated with surgical risk.

[0249] Example 12. A surgical system according to any one of Examples 9, wherein the surgical instrument comprises an end effector, and the user response includes causing the end effector to hover over an alert feature.

[0250] Example 13. A surgical system is intended for use in a surgical procedure. The surgical system comprises a surgical instrument configured to treat tissue in a surgical procedure, an imaging device, a display configured to show a live stream of the surgical field of the surgical procedure, the display of which the live stream is captured by the imaging device, and a control module. The control module is configured to detect a surgical risk, assign a severity level to the surgical risk, and determine a display arrangement of the surgical risk based on the severity level of the surgical risk, wherein the display arrangement includes an alert feature.

[0251] Example 14. The surgical system described in Example 13, wherein the alert feature is an alert icon.

[0252] Example 15. The surgical system according to Example 13 or 14, wherein the display arrangement includes overlaying alert features on the corners of the live stream.

[0253] Example 16. A surgical system according to any one of Examples 13-15, wherein the display arrangement includes changing the alert features based on the severity level of the surgical risk.

[0254] Example 17. A surgical system according to any one of Examples 13 to 15, further comprising the display arrangement causing a change in the alert feature in response to a user response to the alert feature.

[0255] Example 18. A surgical system according to any one of Example 13, wherein the surgical instrument comprises an end effector, and a user response triggers a transition of the end effector in the surgical field between an open configuration and a closed configuration.

[0256] Example 19. A surgical system according to any one of Example 13, wherein the modification of the alert feature includes replacing the alert feature with information associated with surgical risk.

[0257] Example 20. A surgical system according to any one of Example 13, wherein the surgical instrument comprises an end effector, and the user response includes causing the end effector to hover over an alert feature.

[0258] While several forms have been shown and described, it is not the applicant's intention to limit or restrict the attached claims to such details. Many modifications, variations, alterations, substitutions, combinations, and equivalents of these forms can be implemented and will be conceived by those skilled in the art without departing from the scope of this disclosure. Furthermore, the structure of each element related to the described form can be alternatively described as a means for providing the function performed by that element. Also, while materials are disclosed with respect to specific components, other materials may be used. Therefore, it should be understood that the above description and the attached claims are intended to cover all such modifications, combinations, and variations as being included within the scope of the disclosed forms. The attached claims are intended to cover all such modifications, variations, alterations, substitutions, alterations, and equivalents.

[0259] The detailed descriptions above have used block diagrams, flowcharts, and / or embodiments to describe various forms of apparatus and / or processes. To the extent that such block diagrams, flowcharts, and / or embodiments include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flowcharts, and / or embodiments can be individually and / or collectively implemented by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some or all of the forms disclosed herein can be equivalently implemented on integrated circuits as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or substantially any combination thereof, and that designing circuits and / or writing software and / or firmware code falls within the scope of the skills of those skilled in the art in light of this disclosure. Furthermore, as will be understood by those skilled in the art, the mechanisms of the subject matter described herein can be distributed in various forms as one or more program products, and the specific forms of the subject matter described herein are applicable regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0260] Instructions used to program logic to implement various disclosed embodiments may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Furthermore, instructions may be distributed over a network or by other computer-readable media. Thus, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but are not limited to floppy diskettes, optical disks, compact disks, read-only memory (CD-ROMs), and magneto-optical disks, read-only memory (ROMs), random access memory (RAMs), erasable programmable read-only memory (EPROMs), electrically erasable programmable read-only memory (EEPROMs), magnetic or optical cards, flash memory, or tangible machine-readable storage used for transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, non-temporary computer-readable media may include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0261] When used in any aspect of this specification, the term "control circuit" can refer to, for example, a hardwired circuit, a programmable circuit (e.g., a computer processor including one or more individual instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), or a field programmable gate array (FPGA)), a state machine circuit, firmware storing instructions executed by a programmable circuit, and any combination thereof. The control circuit can be embodied, collectively or individually, as part of a larger system, such as, for example, an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuit" includes, but is not limited to, an electrical circuit having at least one individual electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application specific integrated circuit, an electrical circuit forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially executes the processes and / or devices described herein, or a microprocessor configured by a computer program that at least partially executes the processes and / or devices described herein), an electrical circuit forming a memory device (e.g., in the form of a random access memory), and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electrical facility). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog form or digital form, or some combination thereof.

[0262] When used in any aspect of this specification, the term "logic" can refer to an application, software, firmware, and / or circuitry configured to perform any of the foregoing operations. The software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. The firmware may be embodied as code, instructions, or instruction sets within a memory device, and / or hard-coded (e.g., non-volatile) data.

[0263] When used in any aspect of this specification, terms such as "component", "system", "module", etc. can refer to any of a control circuit, computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0264] When used in any aspect of this specification, an "algorithm" refers to a self-collision-free sequence of steps leading to a desired result, and a "step" refers to an operation of a physical quantity and / or logical state that, although not necessarily required, can take the form of an electrical or magnetic signal capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities and are merely convenient labels applied to these quantities and / or states.

[0265] A packet-switched network is one example of a network. Communication devices can communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol is the Ethernet communication protocol, which can enable communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of the Ethernet standard. Alternatively or additionally, communication devices can communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards published by the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, communication devices can communicate with each other using the Frame Relay communication protocol. The Frame Relay communication protocol conforms to or may be compatible with standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, transceivers may communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol conforms to or may be compatible with the ATM standard and / or later versions of this standard, published by the ATM Forum in August 2001 under the title "ATM-MPLS Network Interworking 2.0". Naturally, different and / or later developed connection-oriented network communication protocols are equally construed herein.

[0266] Unless otherwise explicitly stated, as is evident from the foregoing disclosures, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the foregoing disclosures should be understood to refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data represented as physical (electronic) quantities in the registers and memory of a computer system into other data similarly represented as physical quantities in the memory or registers of a computer system or other such information storage, transmission, or display device.

[0267] One or more components may be referred to herein as “configured to,” “configurable to,” “operable / operative to,” “adapted / adaptable,” “able to,” “conformable / conformed to,” and so on. Those skilled in the art will understand that “configured to” generally encompasses active components and / or inactive components and / or standby components, unless the context should interpret it otherwise.

[0268] The terms “proximal” and “distal” are used herein in reference to the clinician operating the handle portion of a surgical instrument. “Proximal” refers to the part closest to the clinician, and “distal” refers to the part further away from the clinician. For convenience and clarity, spatial terms such as “vertical,” “horizontal,” “up,” and “down” may be used herein in reference to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be restrictive and / or absolute.

[0269] Those skilled in the art will generally understand that the terms used herein, and especially in the appended claims (e.g., the text of the appended claims), are generally intended to be "open" terms (for example, the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," and the term "includes" should be interpreted as "includes but is not limited to"). Furthermore, those skilled in the art will understand that if a particular number is intended in an introduced claim recitation, such intent is clearly stated in the claim, and if such statement is not present, such intent does not exist. For example, to aid understanding, subsequent appended claims may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be interpreted as suggesting that any particular claim containing such introduced claim description is limited to claims containing only one such description, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" (for example, "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"). The same applies when introducing a claim description using a definite article.

[0270] In addition, even if a specific number is explicitly stated in the introduced claim, it will be recognized by those skilled in the art that such a statement should typically be interpreted as meaning at least the number stated (for example, if there is a statement that is simply “two descriptions” without any other modifiers, it generally means at least two descriptions, or two or more descriptions). Furthermore, when a notation similar to “at least one of A, B, and C, etc.” is used, such a notation is generally intended to be understood in a way that those skilled in the art will understand (for example, “a system having at least one of A, B, and C” is not limited to systems having only A, only B, only C, both A and B, both A and C, both B and C and / or all of A, B and C, etc.). When expressions similar to "at least one of A, B, or C" are used, such expressions are generally intended to be understood in a way that a person skilled in the art would understand (for example, "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Furthermore, a person skilled in the art will understand that, typically, any disjunctive word and / or phrase representing two or more selective terms should be understood, whether in the specification, claims, or drawings, as intended to include the possibility of including one of those terms, any of those terms, or both of those terms, unless the context requires a different interpretation. For example, the phrase "A or B" will typically be understood to include the possibility of "A" or "B" or "A and B".

[0271] With respect to the attached claims, those skilled in the art will understand that the operations cited herein may generally be performed in any order. Furthermore, while various operations are shown in sequence(s), it should be understood that the operations may be performed in any order other than those shown, or simultaneously. Examples of such alternative orderings may include repetition, alternation, interruption, reordering, augmentation, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context should imply otherwise. Moreover, terms such as “responsive to,” “related to,” or other past tense adjectives are generally not intended to exclude such variations, unless the context should imply otherwise.

[0272] It is worth noting that any reference to “one aspect,” “aspect,” “example,” or “example” means that the specific feature, structure, or characteristic described in relation to that aspect is included in at least one aspect. Therefore, the phrases “in one aspect,” “in aspect,” “example,” and “example” found in various places throughout this specification do not necessarily all refer to the same aspect. Furthermore, specific features, structures, or characteristics can be combined in any preferred manner in one or more aspects.

[0273] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material does not conflict with this Specified. Disclosures expressly stated herein, both in themselves and to the extent required, shall supersede any conflicting statements incorporated herein by reference. Any material, or any part thereof, that is referred to as being incorporated herein by reference but conflicts with current definitions, views, or other disclosures contained herein shall be incorporated only to the extent that there is no conflict between the incorporated material and the current disclosures.

[0274] In summary, the numerous benefits that can be obtained as a result of using the concepts described herein have been described. The above descriptions of one or more forms are presented for illustrative and explanatory purposes only. They are not intended to be comprehensive or to be limited to the exact forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms have been selected and described to illustrate the principle and practical applications, thereby enabling a person skilled in the art to utilize the various forms, along with various modifications, for specific conceivable uses. The claims presented herein are intended to define the overall scope.

[0275] [Implementation Method] (1) A surgical system for use in surgical procedures, A surgical instrument configured to treat tissue in the aforementioned surgical procedure, Imaging device and A display configured to show a live stream of the surgical field of the surgical procedure, wherein the live stream is captured by the imaging device, A control module, To detect surgical risks, Assigning display priority to the aforementioned surgical risks, A surgical system comprising a control module configured to determine the display arrangement of the surgical risk based on the display priority, wherein the display arrangement includes overlaying alert features on the live stream. (2) The surgical system according to Embodiment 1, wherein the alert feature is an alert icon. (3) The surgical system according to Embodiment 1, wherein the display arrangement includes overlaying the alert feature on the corner of the live stream. (4) The surgical system according to Embodiment 1, wherein the display arrangement includes changing the alert feature based on the severity level of the surgical risk. (5) The surgical system according to embodiment 1, wherein the surgical risk is related to the level of bleeding of the tissue.

[0276] (6) The surgical system according to embodiment 1, wherein the surgical risk is associated with tissue parameters. (7) The surgical system according to embodiment 1, wherein the surgical risk is associated with parameters of the surgical instrument. (8) The surgical system according to embodiment 1, wherein the surgical risk is associated with the movement of the tissue. (9) The surgical system according to embodiment 1, further comprising causing a change in the alert feature in response to a user reaction to the alert feature. (10) The surgical system according to embodiment 9, wherein the surgical instrument comprises an end effector, and the user reaction comprises causing a transition of the end effector in the surgical field between an open configuration and a closed configuration.

[0277] (11) The surgical system according to embodiment 9, wherein the change in the alert feature comprises replacing the alert feature with information associated with the surgical risk. (12) The surgical system according to embodiment 9, wherein the surgical instrument comprises an end effector, and the user reaction comprises hovering the end effector over the alert feature. (13) A surgical system for use in a surgical procedure, a surgical instrument configured to treat tissue in the surgical procedure, <00​​​​​​​​​​A surgical system comprising a control module configured to determine a display arrangement of the surgical risk based on the severity level of the surgical risk, wherein the display arrangement includes an alert feature. (14) The surgical system according to Embodiment 13, wherein the alert feature is an alert icon. (15) The surgical system according to Embodiment 13, wherein the display arrangement includes overlaying the alert feature on the corner of the live stream.

[0278] (16) The surgical system according to Embodiment 13, wherein the display arrangement includes changing the alert feature based on the severity level of the surgical risk. (17) The surgical system according to Embodiment 13, further comprising the display arrangement causing a change in the alert feature in response to a user response to the alert feature. (18) The surgical system according to Embodiment 17, wherein the surgical instrument comprises an end effector, and the user response causes a transition of the end effector in the surgical field between an open configuration and a closed configuration. (19) The surgical system according to Embodiment 17, wherein the modification of the alert feature includes replacing the alert feature with information associated with the surgical risk. (20) The surgical system according to embodiment 17, wherein the surgical instrument comprises an end effector, and the user response includes causing the end effector to hover over the alert feature.

Claims

1. A surgical system for use in surgical procedures, A surgical instrument configured to treat tissue in the aforementioned surgical procedure, Imaging device and A display configured to show a live stream of the surgical field of the surgical procedure, wherein the live stream is captured by the imaging device, A control module, To detect surgical risks, Assigning display priority to the aforementioned surgical risks, A surgical system comprising a control module configured to determine the display arrangement of the surgical risk based on the display priority, wherein the display arrangement includes overlaying alert features on the live stream and causing a change to the alert features in response to a user response to the alert features.

2. The surgical system according to claim 1, wherein the alert feature is an alert icon.

3. The surgical system according to claim 1, wherein the display arrangement includes overlaying the alert feature on the corner of the live stream.

4. The surgical system according to claim 1, wherein the display arrangement includes changing the alert features based on the severity level of the surgical risk.

5. The surgical system according to claim 1, wherein the surgical risk is related to the level of bleeding in the tissue.

6. The surgical system according to claim 1, wherein the surgical risk is associated with tissue parameters.

7. The surgical system according to claim 1, wherein the surgical risk is associated with the parameters of the surgical instrument.

8. The surgical system according to claim 1, wherein the surgical risk is associated with the movement of the tissue.

9. The surgical system according to claim 1, wherein the surgical instrument comprises an end effector, and the user response causes a transition of the end effector in the surgical field between an open configuration and a closed configuration.

10. The surgical system according to claim 1, wherein the modification of the alert feature includes replacing the alert feature with information associated with the surgical risk.

11. The surgical system according to claim 1, wherein the surgical instrument comprises an end effector, and the user response includes causing the end effector to hover over the alert feature.

12. A surgical system for use in surgical procedures, A surgical instrument configured to treat tissue in the aforementioned surgical procedure, Imaging device and A display configured to show a live stream of the surgical field of the surgical procedure, wherein the live stream is captured by the imaging device, A control module, To detect surgical risks, Assigning severity levels to the aforementioned surgical risks, A surgical system comprising a control module configured to determine a display arrangement of the surgical risk based on the severity level of the surgical risk, wherein the display arrangement includes an alert feature, and the display arrangement further includes causing a change in the alert feature in response to a user response to the alert feature.

13. The surgical system according to claim 12, wherein the alert feature is an alert icon.

14. The surgical system according to claim 12, wherein the display arrangement includes overlaying the alert feature on the corner of the live stream.

15. The surgical system according to claim 12, wherein the display arrangement includes changing the alert features based on the severity level of the surgical risk.

16. The surgical system according to claim 12, wherein the surgical instrument comprises an end effector, and the user response causes a transition of the end effector in the surgical field between an open configuration and a closed configuration.

17. The surgical system according to claim 12, wherein the modification of the alert feature includes replacing the alert feature with information associated with the surgical risk.

18. The surgical system according to claim 12, wherein the surgical instrument comprises an end effector, and the user response includes causing the end effector to hover over the alert feature.

Images