Surgical system with multi-modality image display

Abstract

To provide a method for producing a multi-modality image of a surgical scene.SOLUTION: The method comprises: capturing light reflected from the surgical scene; producing first image information corresponding to a first modality image; producing second image information corresponding to a second modality image; selecting a portion of the second modality image based at least in part upon anatomical structure information included within the selected portion; and producing simultaneously within a display at least a portion of the first modality image of the surgical scene and the selected portion of the second modality image.SELECTED DRAWING: Figure 21A

Description

[Technical Field]

[0001] (Priority Claim) This application claims the benefit of priority to U.S. Patent Application No. 62 / 421,095, filed November 11, 2016, which is incorporated herein by reference in its entirety.

[0002] This application relates to a surgical system with multi-modality image display. [Background technology]

[0003] Image-guided surgery techniques provide surgeons with the ability to visualize the internal structures of anatomical objects within a surgical scene. Improved visualization capabilities allow surgeons to better perceive the actual location of surgical instruments and their physical relationships to important or hidden anatomical structures within the anatomical object, which can result in safer and more effective minimally invasive surgical procedures. Summary of the Invention

[0004] The following summary introduces certain aspects of the inventive subject matter in order to provide a basic understanding. This summary is not an extensive overview of the inventive subject matter and is not intended to identify key or critical elements or to delineate the scope of the inventive subject matter. While this summary includes information related to various aspects and embodiments of the inventive subject matter, its sole purpose is to present some aspects and embodiments in a general form as a prelude to the more detailed description that follows.

[0005] In one aspect, a method for generating a multi-modality image of a surgical scene is provided. A camera captures light reflected from the surgical scene. First image information corresponding to a first modality image of the surgical scene is generated. Second modality image information corresponding to the surgical scene is generated. A portion of the second modality image is selected based at least in part on anatomical structure information it contains. A computer is configured to use the first image information and the second image information to simultaneously generate at least a portion of the first modality image of the surgical scene and a selected portion of the second modality image of the surgical scene within a display.

[0006] In another aspect, a method for generating a multi-modality image of a surgical scene is provided. A camera captures light reflected from the surgical scene. First image information corresponding to a first modality image of the surgical scene is generated. Second image information corresponding to a second modality image of the surgical scene is generated. A focus position of a user's eye is tracked within the display. A portion of the second modality image is selected based at least in part on the tracked focus of the user's eye. A computer is configured to use the first image information and the second image information to simultaneously generate within the display at least a portion of the first modality image of the surgical scene and a selected portion of the second modality image of the surgical scene.

[0007] In yet another aspect, a method for generating a multi-modality image of a surgical scene is provided. A camera captures light reflected from the surgical scene. First image information corresponding to a first modality image of the surgical scene is generated. Second image information corresponding to a second modality image of the surgical scene is generated. Surgical stages are tracked during the surgical procedure. A modality image format is selected based at least in part on the tracked surgical stages. A computer is configured to generate at least a portion of the first modality image and at least a portion of the second modality image within a display in the selected format using the first image information and the second image information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a plan view of a minimally invasive teleoperated surgical system. [Figure 2] FIG. 1 is a perspective view of a surgeon's console. [Figure 3] FIG. 1 is a perspective view of an electronics cart. [Figure 4] FIG. 1 is a schematic diagram of a teleoperated surgical system. [Figure 5A] 1 is an exemplary diagram of a teleoperated surgical system. [Figure 5B] FIG. 1 is a perspective view of a patient side cart of a surgical system. [Figure 5C] FIG. 1 is an illustration depicting a surgical scene and also showing a camera used to record the scene and an endoscope fitted with an optical filter for filtering light wavelengths, according to some embodiments. [Figure 6] FIG. 1 is an elevational view of a surgical instrument. [Figure 7] FIG. 1 is a perspective view of an instrument manipulator. [Figure 8] FIG. 1 is a diagram of a surgical planning tool. [Figure 9] FIG. 1 is an exemplary diagram showing a surgical information atlas stored in a non-transitory storage device, according to some embodiments. [Figure 10] 1 is an exemplary diagram illustrating certain details of an exemplary surgical record information structure stored within a surgical information atlas, according to some embodiments. [Figure 11] 1 is an exemplary diagram showing an example clear image of a surgical scene, according to some embodiments. [Figure 12] 12 is an exemplary diagram representing an exemplary narrow band imaging modality image of the exemplary surgical scene of FIG. 11 in which an image of an anatomical structure within a first anatomical object is highlighted through a narrow band imaging (NBI) display modality. [Figure 13]12 is an exemplary diagram depicting an exemplary Raman spectroscopy modality image of the exemplary surgical scene of FIG. 11 in which an image of an anatomical structure within a second anatomical object is highlighted through the Raman spectroscopy display modality. [Figure 14] 12 is an exemplary diagram depicting an exemplary hyperspectral (HIS) modality image of the exemplary surgical scene of FIG. 11 displayed using a HIS image display modality. [Figure 15] 12 is an exemplary diagram depicting a computed tomography slice display modality of the exemplary surgical scene of FIG. 11 in accordance with some embodiments. [Figure 16] 12 is an exemplary multi-modality display of the surgical scene of FIG. 11 using multiple display modalities combined using a stitch format according to some embodiments. [Figure 17] 12 is an exemplary alternative multi-modality display of the surgical scene of FIG. 11 using multiple display modalities combined using a stitching format according to some embodiments. [Figure 18] 12 is an exemplary multi-modality display of the surgical scene of FIG. 11 using multiple display modalities combined using a picture-in-picture format according to some embodiments. [Figure 19] 12 is an exemplary alternative multi-modality display of the surgical scene of FIG. 11 using multiple display modalities combined using a picture-in-picture format according to some embodiments. [Figure 20] 12 is an exemplary display of the surgical scene of FIG. 11 using information from multiple display modalities using annotation formatting according to some embodiments. [Figure 21A] FIG. 1 is an exemplary flow diagram illustrating a first process for dynamically presenting multi-modality surgical images according to some embodiments. [Figure 21B] 1A-1C are exemplary diagrams illustrating respective first image information stored in non-transitory storage devices that can be used to generate different image modality images. [Figure 21C] 10A-10C are exemplary diagrams depicting respective second image portion information stored in non-transitory storage that can be used to generate different imaging modality image portions. [Figure 22] FIG. 10 is an exemplary flow diagram detailing a sub-process for selecting a display modality and portions of the selected display modality according to some embodiments. [Figure 23] FIG. 10 is an exemplary flow diagram detailing a sub-process for receiving a user selection of an imaging modality according to some embodiments. [Figure 24] FIG. 10 is an exemplary diagram illustrating an example modality selection user interface UI displayed within a viewer of a surgical system according to some embodiments. [Figure 25A] FIG. 1 is an exemplary diagram illustrating two-dimensional eye tracking according to some embodiments. [Figure 25B] FIG. 1 is an exemplary diagram illustrating two-dimensional eye tracking according to some embodiments. [Figure 25C] FIG. 1 is an exemplary diagram illustrating depth eye tracking according to some embodiments. [Figure 26] FIG. 10 is an exemplary flow diagram detailing a sub-process for selecting a multi-modality display format according to some embodiments. [Figure 27] FIG. 10 is an exemplary diagram showing an exemplary format selection user interface UI displayed within a viewer of a surgical system according to some embodiments. [Figure 28] FIG. 10 is an exemplary flow diagram depicting specific details of a sub-process for determining whether surgical stage information and corresponding surgical stage rules together indicate a display modality according to some embodiments. [Figure 29] 1 is an exemplary diagram depicting exemplary surgical stage signatures and corresponding modalities according to some embodiments. FIG. [Figure 30]FIG. 10 is an exemplary flow diagram depicting certain details of a sub-process for generating an imaging modality portion according to some embodiments. [Figure 31] 1 is an exemplary diagram depicting exemplary image signatures and corresponding annotations according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0009] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting, but rather define the invention protected by the claims. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the invention. Like numbers in two or more figures represent the same or similar elements.

[0010] The following description is presented to enable those skilled in the art to create and use multi-modality images in a surgical system. Various modifications to the present embodiment will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the inventive subject matter. Moreover, in the following description, numerous details are set forth for purposes of explanation. However, those skilled in the art will understand that the inventive subject matter may be practiced without these specific details. In other instances, well-known machine components, processes, and data structures are shown in block diagram form so as not to obscure the present disclosure with unnecessary detail. The same reference numbers may be used to represent different views of the same items in different drawings. Flow diagrams in the drawings referenced below are used to represent processes. A computer system may be configured to perform some of these processes. Blocks in the flow diagrams representing computer-implemented processes represent the configuration of a computer system with computer program code that performs the operations described with reference to those modules. Thus, the inventive subject matter is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0011] Minimally invasive remote surgery system Teleoperation refers to the operation of machinery at a distance. In minimally invasive teleoperated medical systems, surgeons can use endoscopes that include cameras to view the surgical site inside the patient's body. In some embodiments, stereoscopic images can be captured, which allows for depth perception during the surgical procedure.

[0012] FIG. 1 is an exemplary plan view of a minimally invasive teleoperated surgical system 10 typically used to perform a minimally invasive diagnostic or surgical procedure on a patient 12 reclining on an operating table 14. The system includes a surgeon's console 16 for use by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The minimally invasive teleoperated surgical system 10 further includes patient side cart(s) 22 and an electronics cart 24. The patient side cart 22 can manipulate at least one surgical instrument 26 through a minimally invasive incision in the body of the patient 12 while the surgeon 18 views the surgical site through the surgeon's console 16. Images of the surgical site can be acquired by an endoscope 28, such as a stereo endoscope, which can be manipulated by the patient side cart 22 to orient the endoscope 28. A computer processor located on the electronics cart 24 can be used to process images of the surgical site for subsequent display to the surgeon 18 through the surgeon's console 16. It should be noted that while individual system components (i.e., patient side cart 22, electronics cart 24, and surgeon's console 16) are shown and described for illustrative purposes, in various embodiments, the components included therein may be combined and / or separated. For example, in some embodiments, the computer processor of the electronics cart 24 may be incorporated into the surgeon's console 16 and / or the patient side cart 22. The number of surgical instruments 26 used at one time generally depends on, among other factors, the diagnostic or surgical procedure and the spatial constraints within the surgical site. If one or more of the surgical instruments 26 used during a procedure needs to be replaced, an assistant 20 can remove the surgical instrument 26 from the patient side cart 22 and replace it with another surgical instrument 26 from a tray 30 in the operating room.

[0013] FIG. 2 is a perspective view of the surgeon's console 16. The surgeon's console 16 includes a viewer display 31, which includes a left-eye display 32 and a right-eye display 34, for presenting the surgeon 18 with a coordinated stereoscopic view of the surgical site, enabling depth perception. The console 16 also includes one or more control inputs 36. One or more surgical instruments mounted for use on the patient side cart 22 (shown in FIG. 1) move in response to manipulation of the one or more control inputs 36 by the surgeon 18. The control inputs 36 may provide the same mechanical degrees of freedom as the associated surgical instrument 26 (shown in FIG. 1) to give the surgeon 18 a sense of telepresence, or a sense that the control inputs 36 are integral with the instrument 26, such that the surgeon has a strong sense of direct control over the instrument 26. To this end, position, force, and tactile feedback sensors (not shown) may be used to communicate position, force, and tactile perception from the surgical instrument 26 through the control inputs 36 and back to the surgeon's hand, subject to communication delays. It should be noted that while a physical console 16 with a fixed viewer 31 and mechanically coupled control inputs 36 is shown and described for illustrative purposes, in various other embodiments, an "ungrounded" control input and / or display structure may be used. For example, in some embodiments, the viewer 31 may be a head-mounted display and / or the control inputs 36 may be mechanically independent of any base structure (e.g., wired, wireless, or gesture-based, such as Kinect from Microsoft).

[0014] The surgeon's console 16 is typically located in the same room as the patient so that the surgeon can directly monitor the procedure and, if necessary, be physically present and speak to the patient's assistant directly rather than by telephone or other communication medium. However, the surgeon can be located in a different room, an entirely different building, or even remotely from the patient, allowing for remote surgical procedures.

[0015] FIG. 3 is a perspective view of the electronics cart 24. The electronics cart 24 can be coupled to the endoscope 28 and includes a computer processor for processing captured images for subsequent display, such as to the surgeon at the surgeon's console or to another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the computer processor on the electronics cart 24 can process the captured images to present the surgeon with a coordinated stereoscopic image of the surgical site. Such adjustments can include alignment between opposing images and adjusting the stereoscopic working distance of the stereoscopic endoscope. As another example, the image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture devices, such as optical aberrations. Optionally, the devices in the electronics cart can be integrated into the surgeon's console or patient side cart, or distributed to various other locations within the operating room.

[0016] FIG. 4 schematically illustrates a teleoperated surgical system 50 (such as the minimally invasive teleoperated surgical system 10 of FIG. 1). A surgeon's console 52 (such as the surgeon's console 16 in FIG. 1) can be used by a surgeon to control a patient side cart 54 (such as the patient side cart 22 in FIG. 1) during a minimally invasive procedure. The patient side cart 54 can use an imaging device, such as a stereo endoscope, to capture images of the surgical site and output the captured images to a computer processor located on an electronics cart 56 (such as the electronics cart 24 in FIG. 1). The computer processor typically includes one or more data processing boards designed to execute computer-readable code stored in a non-volatile memory device of the computer processor. In one embodiment, the computer processor can process the captured images in various ways prior to subsequent display. For example, the computer processor can overlay the captured images with a virtual control interface before displaying a composite image to the surgeon via the surgeon's console 52.

[0017] Additionally or alternatively, the captured images may undergo image processing by a computer processor located outside the electronics cart 56. In one embodiment, the teleoperated surgical system 50 includes an optional computer processor (CPU) 58 similar to the computer processor located in the electronics cart 56, and the patient side cart 54 outputs the captured images to the computer processor (CPU) 58 for image processing before display on the surgeon's console 52. In another embodiment, the captured images first undergo image processing by the computer processor of the electronics cart 56 and then undergo additional image processing by the computer processor 58 before display on the surgeon's console 52. The teleoperated surgical system 50 may include an optional display 60, as indicated by the dashed line. The display 60 is coupled to a computer located in the electronics cart 56 and the computer processor (CPU) 58, and the captured images processed by these computer processors may be displayed on the display 60 in addition to being displayed on the display of the surgeon's console 52.

[0018] FIG. 5A is an exemplary simplified block diagram illustrating the arrangement of components of a teleoperated surgical system 10 for performing a surgical procedure using one or more mechanical support arms 510, according to some embodiments. Aspects of the system 10 include the ability to operate robotically and autonomously. These mechanical support arms 510 often support surgical instruments. For example, a mechanical surgical arm (e.g., a central mechanical surgical arm 510C) may be used to support an endoscope equipped with a stereo or three-dimensional surgical image capture device 101C. The mechanical surgical arm 510C may include a sterile adapter or clamp, clip, screw, slot / groove, or other fastening mechanism for mechanically securing the endoscope, including the image capture device 101C, to the mechanical arm.

[0019] A user or operator O (typically a surgeon) performs a surgical procedure on a patient P by manipulating control input devices 36, such as handgrips and foot pedals, on the master control console 16. The operator can view video frames of images of the surgical site inside the patient's body through a stereoscopic viewer 31. A computer processor (CPU) 58 in the console 16 directs the movement of the remotely controlled endoscopic surgical instruments 101A-101C via control lines 159, effecting instrument motion using a patient-side system 24 (also referred to as a patient-side cart).

[0020] The patient-side system 24 includes one or more mechanical support arms 510. Typically, the patient-side system 24 includes at least three mechanical surgical arms 510A-510C (commonly referred to as mechanical surgical support arms 510) supported by corresponding positioning setup arms 156. The central mechanical surgical arm 510C may support an endoscopic camera 101C suitable for capturing images within the camera's field of view. The central left and right mechanical surgical support arms 510A and 510B may support instruments 101A and 101B, respectively, capable of manipulating tissue.

[0021] FIG. 5B is a perspective view of a patient side cart 500 of the minimally invasive teleoperated surgical system 10, according to an embodiment. The patient side cart 500 includes one or more support arm assemblies 510. A surgical instrument manipulator 512 is attached to the end of each support arm assembly 510. Additionally, each support arm assembly 510 can optionally include one or more setup joints (e.g., unpowered and / or lockable) used to position the attached surgical instrument manipulator 512 relative to the patient for surgery. As shown, the patient side cart 500 rests on the floor. In other embodiments, the working portion of the patient side cart can be attached to a wall, ceiling, an operating table 526 that also supports the patient's body 522, or other operating room equipment. Additionally, while the patient side cart 500 is shown as including four surgical instrument manipulators 512, more or fewer surgical instrument manipulators 512 may be used.

[0022] A functional teleoperated surgical system generally includes a vision system portion that allows a user of the teleoperated surgical system to view the surgical site from outside the patient's body 522. The vision system typically includes a camera instrument 528 for capturing video images and one or more video displays for displaying the captured video images. In some surgical system configurations, the camera instrument 528 includes optics that transmit images from the distal end of the camera instrument 528 to one or more image sensors (e.g., CCD or CMOS sensors) that generate digital image information outside the patient's body 522. Alternatively, the imaging sensor(s) can be located at the distal end of the camera instrument 528, and the digital image information signals generated by the sensor(s) can be transmitted along leads or wirelessly for processing and display on one or more video displays. One example of a video display is the stereoscopic display on the surgeon's console in a surgical system commercialized by Intuitive Surgical, Inc. of Sunnyvale, California.

[0023] 5A-5B, attached to each surgical instrument manipulator 512 is a surgical instrument 520 that operates at a surgical site within a patient's body 522. Each surgical instrument manipulator 512 can be provided in a variety of forms that allow the associated surgical instrument to move with one or more mechanical degrees of freedom (e.g., six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints constrain each manipulator 512 to move its associated surgical instrument about the instrument's center of motion, which remains stationary relative to the patient, and which is typically located at the point where the instrument enters the body.

[0024] In one embodiment, the surgical instrument 520 is controlled by computer-assisted teleoperation. A functional minimally invasive teleoperated surgical system includes a control input that receives input from a user of the teleoperated surgical system (e.g., a surgeon or other medical professional). The control input is in communication with one or more computer-controlled teleoperated actuators, e.g., one or more motors to which the surgical instrument 520 is coupled. In this manner, the surgical instrument 520 moves in response to the medical professional's movement of the control input. In one embodiment, the one or more control inputs are included in a surgeon's console, such as the surgeon's console 16 shown in FIG. 2. The surgeon can operate the control input device 36 of the surgeon's console 16 to operate the teleoperated actuators of the patient side cart 500. Forces generated by the teleoperated actuators are transmitted via a drivetrain mechanism that transfers the force from the teleoperated actuators to the surgical instrument 520.

[0025] 5A-5B , in one embodiment, a surgical instrument 520 and a cannula 524 are removably coupled to a manipulator 512, and the surgical instrument 520 is inserted through the cannula 524. One or more teleoperated actuators of the manipulator 512 move the entire surgical instrument 512. The manipulator 512 further includes an instrument carriage 530. The surgical instrument 520 is removably connected to the instrument carriage 530. In one embodiment, the instrument carriage 530 houses one or more teleoperated actuators therein that provide several controller actions that the surgical instrument 520 translates into various movements of the end effector of the surgical instrument 520. Thus, the teleoperated actuators in the instrument carriage 530 move only one or more components of the surgical instrument 520, and not the entire instrument. Inputs for controlling either the entire instrument or the instrument components are such that inputs provided by a surgeon or other medical personnel to a control input (a “master” command) are translated into corresponding actions (a “slave” response) by the surgical instrument.

[0026] According to some embodiments, the surgical system 10 can have multiple system operating states, including docked, following, instrument type, and head-in. During the docked system state, one or more manipulators 512 are coupled to the cannula 524. During the following system state, the surgical instruments ("slaves") are tracking the control inputs ("master" commands). During the instrument type system state, the system has installed therein a set of instruments suitable for performing a particular surgical procedure or for performing a particular surgical task during a surgical procedure. During the head-in system state, the system waits for the surgeon to indicate that he or she has held the "master" control input device.

[0027] In alternative embodiments, the instrument carriage 530 does not house a teleoperated actuator. The teleoperated actuators enabling various movements of the end effectors of the surgical instrument 520 are housed at a location remote from the instrument carriage 530, such as elsewhere on the patient side cart 500. A cable-based force transmission mechanism or the like is used to transfer the movement of each remotely located teleoperated actuator to a corresponding instrument interface actuator output located on the instrument carriage 530. In some embodiments, the surgical instrument 520 is mechanically coupled to a first actuator that controls a first movement of the surgical instrument, such as longitudinal (z-axis) rotation. The surgical instrument 520 is mechanically coupled to a second actuator that controls a second movement of the surgical instrument, such as two-dimensional (x, y) movement. The surgical instrument 520 is mechanically coupled to a third actuator that controls a third movement of the surgical instrument, such as opening and closing of the jaw end effectors.

[0028] 5C is an exemplary diagram depicting a surgical scene 550 and also showing an endoscope 101C equipped with a camera 528 used to record the scene according to some embodiments. The scene 550 is positioned within a patient's body cavity. The scene 550 includes an exemplary virtual spherical anatomical structure 552 including a geometric contour 554. The scene 550 encompasses a surgical instrument 556. First and second cameras 528 and 533 are attached to the endoscope 101C to capture the scene, which is displayed in the viewer 31 and recorded for later playback. A selectable first optical filter 528 is associated with the first camera. First and second light sources 531 are provided. A selectable second filter 529 is associated with at least one of the light sources 531.

[0029] FIG. 6 is a side view of a surgical instrument 520 including a distal portion 650 and a proximal control mechanism 640 coupled by an elongated tube 610 having an elongated tube central axis 611. The surgical instrument 520 is configured to be inserted into a patient's body and used to perform a surgical or diagnostic procedure. The distal portion 650 of the surgical instrument 520 can provide any of a variety of end effectors 654, such as the illustrated forceps, needle drivers, cauterization devices, cutting tools, imaging devices (e.g., endoscopes or ultrasound probes), etc. The surgical end effector 654 can include functional mechanical degrees of freedom, such as jaws that open and close, or a knife that moves along a path. In the illustrated embodiment, the end effector 654 is coupled to the elongated tube 610 by a wrist 652 that allows the end effector to be oriented relative to the elongated tube central axis 611. The surgical instrument 520 can also include stored information (e.g., on a semiconductor memory associated with the instrument), which can be permanent or updatable by a surgical system configured to operate the surgical instrument 520. Thus, the surgical system may provide for one-way or two-way communication of information between the surgical instrument 520 and one or more components of the surgical system.

[0030] FIG. 7 is a perspective view of a surgical instrument manipulator 512. The instrument manipulator 512 is shown without a surgical instrument attached. The instrument manipulator 512 includes an instrument carriage 530 to which a surgical instrument (e.g., surgical instrument 520) can be removably connected. The instrument carriage 530 houses multiple teleoperated actuators. Each teleoperated actuator includes an actuator output 705. When a surgical instrument is mounted on the instrument manipulator 512, one or more instrument inputs (not shown) of an instrument proximal control mechanism (e.g., proximal control mechanism 640 in FIG. 6) are mechanically coupled to a corresponding actuator output 705. In one aspect, this mechanical coupling is direct, with the actuator output 705 in direct contact with the corresponding instrument input. In another aspect, this mechanical coupling occurs through an intermediate interface, such as a component of a drape configured to provide a sterile barrier between the instrument manipulator 512 and the associated surgical instrument.

[0031] In one embodiment, actuation of one or more instrument inputs by corresponding teleoperated actuators results in movement of mechanical degrees of freedom of the surgical instrument. For example, in one embodiment, the surgical instrument attached to instrument manipulator 512 is surgical instrument 520 shown in FIG. 6. Referring to FIG. 6, in one embodiment, movement of one or more instrument inputs of proximal control mechanism 640 by corresponding teleoperated actuators rotates elongate tube 610 (and attached wrist 652 and end effector 654) relative to proximal control mechanism 640 about elongate tube centerline 611. In another embodiment, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of wrist 652, which orients end effector 654 relative to elongate tube central axis 611. In another embodiment, movement of one or more instrument inputs by corresponding teleoperated actuators results in movement of one or more movable elements (e.g., jaw members, knife members, etc.) of end effector 654. Thus, surgical instruments with various mechanical degrees of freedom mounted on instrument manipulator 512 can be moved by operation of remotely controlled actuators on instrument carriage 530 .

[0032] Annotate recorded video FIG. 8 shows a schematic diagram of an exemplary surgical planning tool 800. In one embodiment, the surgical planning tool 800 includes a teleoperated surgery system 850 in data communication with an electronic medical device records database 830. The teleoperated surgery system 850 shown here is similar to the teleoperated surgery system 850 shown in FIG. 4. In one embodiment, the electronic medical record database 830 contains medical records of patients treated at a particular hospital or multiple hospitals. The database 830 can be implemented on a server located on-site at the hospital. The medical record entries contained in the database 830 can be accessed from a hospital computer over an intranet network. Alternatively, the database 830 can be implemented on a remote server located off-site from the hospital, for example, using one of several cloud data storage services. In this case, the medical record entries in the database 830 are stored on the cloud server and can be accessed by a computer with internet access.

[0033] In one embodiment, a surgical procedure is performed on a first patient using teleoperated surgical system 850. An imaging device associated with teleoperated surgical system 850 captures images of the surgical site and displays the captured images as frames of video on the display of surgeon's console 52. In one embodiment, a medical professional at surgeon's console 52 uses an input device on surgeon's console 52 to highlight or annotate specific patient anatomical structures shown in the displayed video. One example of such an input device is control input 36 shown in FIG. 2, which is coupled to a cursor that operates in conjunction with a graphic user interface overlaid on the displayed video. The graphic user interface may include a QWERTY keyboard, a pointing device such as a mouse and interactive screen display, a touchscreen display, or other means for data or text entry or voice annotation and / or text conversion via a microphone and processor. Thus, the medical professional may highlight specific structures of interest in the displayed image or enter text annotations.

[0034] In one embodiment, the surgical site video is additionally displayed on a display located on the electronics cart 56. In one embodiment, the electronics cart display is a touchscreen user interface that can be used by the clinician to highlight and annotate specific portions of the patient's anatomy shown on the image displayed for display on the electronics cart display. The user can highlight portions of the displayed image by touching portions of the patient's anatomy displayed on the touchscreen user interface. Additionally, a graphical interface including a QWERTY keyboard can be overlaid on the displayed image. The user can use the QWERTY keyboard to enter text annotations.

[0035] In one embodiment, surgical site videos captured by imaging devices associated with teleoperated surgery system 850 are recorded by teleoperated surgery system 850 and stored in database 830 in addition to being displayed to the user in real time or near real time. Highlights and / or annotations associated with the recorded videos made by the user can also be stored in database 830. In one embodiment, highlights made by the user are embedded in the recorded videos before storing them in database 830. The recorded videos can then be retrieved for viewing. In one embodiment, a viewer of the recorded videos can select whether highlights are displayed or hidden from view. Similarly, annotations associated with the recorded videos can also be stored in database 830. In one embodiment, annotations made by the user are used to tag the recorded videos and can be used to provide a means of identifying the subject matter contained in the recorded videos. For example, one annotation may describe the conditions of a particular disease state. This annotation is used to tag the recorded videos. Later, a person wanting to view recorded procedures related to this condition can locate the videos using a keyword search.

[0036] Searching for saved videos In some cases, it may be desirable for a medical practitioner to be able to view video recordings of previous surgical procedures performed on a given patient. In one embodiment, a patient who previously underwent a first surgical procedure to treat a medical condition subsequently requires a second surgical procedure to treat a recurrence of the same medical condition or to treat an anatomical structure located near the surgical site of the first surgical procedure. In one embodiment, surgical site events of the first surgical procedure were captured in a surgical site video recording, and the video recording was archived in database 830 as part of the patient's electronic medical record. Before performing the second surgical procedure on the patient, the medical practitioner can perform a search of database 830 to find video recordings of the patient's previous surgical procedures.

[0037] In some cases, it may be desirable for a medical practitioner planning to perform a surgical procedure on a patient to be able to view video recordings of similar surgical procedures performed on individuals with certain characteristics similar to the patient. In one embodiment, surgical site video recordings of a surgical procedure can be tagged with metadata information, such as the patient's age, gender, body mass index, genetic information, and the type of procedure the patient underwent, before each video recording is archived in database 830. In one embodiment, the metadata information used to tag the video recordings is automatically retrieved from the patient's then-existing medical records and then used to tag the video recordings before they are archived in database 830. Thus, before performing a medical procedure on a patient, the medical practitioner can search database 830 for video recordings of similar procedures performed on patients who share certain characteristics in common with the patient. For example, if the medical practitioner plans to use teleoperated surgery system 850 to perform a prostatectomy on a 65-year-old male patient with a high body mass index, the medical practitioner can search database 830 for surgical site video recordings of prostatectomies performed using teleoperated surgery system 850 on other men of a similar age and with a similarly high body mass index.

[0038] In one embodiment, the video recording of the surgical procedure is transmitted by database 830 to optional personal computer 820 (shown in dashed lines) and made available for review by the medical personnel scheduled to perform the surgical procedure. Additionally or alternatively, video recordings of previous surgical procedures can be transmitted by database 830 to teleoperated surgical system 850 and made available for pre- or intra-operative review. In one embodiment, the video recordings are displayed by teleoperated surgical system 850 on a display located on surgeon's console 52. In another embodiment, the video recording of the first surgical procedure is displayed on a display located on electronics cart 56.

[0039] Cloud-based video database In one embodiment, database 830 is implemented on a remote server using a cloud data storage service and is accessible by multiple healthcare providers. Referring to FIG. 8 , as indicated by dashed lines, surgical planning tool 800 optionally includes teleoperated surgical system 850 (shown in dashed lines) and personal computer 840 (shown in dashed lines). In one embodiment, teleoperated surgical system 850 is similar to teleoperated surgical system 850 and personal computer 840 is similar to personal computer 820, except that teleoperated surgical system 850 and personal computer 820 are located at a first healthcare provider and teleoperated surgical system 850 and personal computer 840 are located at a second location or second healthcare provider. In one embodiment, a first patient requires surgical treatment for a medical condition and undergoes a surgical procedure at the first healthcare provider using teleoperated surgical system 850. A video recording of the surgical procedure is archived in database 830. Subsequently, a second patient requires surgical treatment for the same medical condition and is scheduled to undergo the surgical treatment at a second healthcare provider using teleoperated surgery system 850. Prior to performing the surgical procedure on the second patient, the medical personnel access database 830 via a secure internet connection and search database 830 for surgical site video recordings of similar procedures. In one embodiment, the medical personnel treating the second patient can retrieve the video recording of the first patient's surgical procedure from database 830 without gaining knowledge of the first patient's identity. In this manner, the first patient's privacy is maintained. In one embodiment, the video recording of the first patient's surgical procedure includes highlights and / or annotations made by the medical personnel who treated the first patient.

[0040] Computer-based pattern matching and analysis The surgical planning tool 800 may include pattern matching and analysis algorithms implemented in the form of computer-executable code. In one embodiment, the pattern matching and analysis algorithms are stored in a non-volatile memory device of the surgical planning tool 800 and configured to analyze video recordings archived in the database 830. As previously described, each video recording archived in the database 830 may be tagged and / or embedded with specific metadata information. This metadata information may include patient information, such as the patient's age, gender, and other information describing the patient's health or medical history. In addition, as previously described, the metadata information may include highlights or annotations made by a medical professional. In one embodiment, these highlights and annotations are embedded in the video recordings and archived with the videos in the database 830.

[0041] In one embodiment, the pattern matching and analysis algorithm includes an image analysis component that identifies patterns of shape and color shared among multiple video recordings stored in database 830. The pattern matching and analysis algorithm then reviews tagged metadata associated with this subset of video recordings to determine whether any words or phrases are frequently associated with videos in this subset. These analyses performed by the pattern matching and analysis algorithm can be used to help clinicians make decisions about patient anatomy, preferred surgical approach, pathology, potential complications, etc.

[0042] Surgical Information Atlas FIG. 9 is an exemplary diagram depicting a surgical information atlas 902 stored in non-transitory storage 904 within the medical record database 830, according to some embodiments. The surgical instrument atlas stores information created in connection with a surgical procedure. For example, images displayed during a surgical procedure can be recorded for storage in the information atlas 902. Some video images may be clear, i.e., unenhanced, some images may include narrow-band imaging (NBI) information, some images may include optical coherence tomography (OCT) information, some images may include Raman spectroscopy information, some images may include fluorescence information, some images may include hyperspectral imaging information, and some images may include CT information. Additionally, information such as surgical instrument kinematic information and surgical stage information, such as surgical system operating states and surgical instrument operating states, is also recorded at different points in the surgical procedure. The surgical system operating state includes, for example, one or more of information related to the kinematic position of the system arm, the position of the cart, and the master control position. Different surgical instruments have different operating states. The operating state of a surgical instrument includes the instrument's kinematics, including, but not limited to, pitch, yaw, roll, and jaw position. During or after surgery, a user, such as a surgeon or other member of the surgical team, can annotate recorded video image information of a surgical scene, such as NBI images, Raman spectroscopy images, OCT images, fluorescence images, or hyperspectral images, with metadata indicating particular anatomical structures of interest, such as blood vessels, tumors, dysplasias, nerves, or connective tissue. The annotations may include, for example, one or more of written notes tagged to the recorded video information and / or coloring or highlighting (e.g., telestration) of images within the video recording, or a combination thereof. The recorded information is used to construct a surgical information atlas 902, which provides associations between surgical image information, surgical stage information, and information related to an instance of a surgical procedure, and this information is stored in a surgical record information structure.

[0043] FIG. 10 is an exemplary diagram showing specific details of an exemplary surgical record information structure 1006 stored within the surgical information atlas 902, according to some embodiments. The patient health record field (HR) 1006-1 provides information about the patient undergoing surgery, such as age, weight, blood type, height, gender, and race. The physician information field (PI) 1006-2 provides information about the surgeon performing the individual surgery, such as their general experience level and their level of experience operating a robotic-assisted surgical system. The surgical system identifier field (SID) 1006-3 provides information about the surgical system used to perform the surgery, such as the manufacturer, model, and serial number. The video image field (MPI) 1006-4 provides information such as video images recorded during surgery. The surgical stage field (SST) 1006-5 provides surgical system operating status information and surgical instrument operating status information recorded during surgery. The non-linear imaging field (NBI) 1006-6 provides information such as video images recorded using NBI during surgery. The Raman field (Ram) 1006-7 provides information such as video images recorded using Raman spectroscopy during surgery. The Fluorescence field (Flu) 1006-8 provides information such as video images recorded using fluorescence during surgery. The HSI field (HSI) 1006-9 provides information such as video images recorded using hyperspectral imaging during surgery. The Surgical Annotation field (Ann) 1006-10 provides annotation information such as descriptive information or expert analysis related to the surgical procedure represented in the information structure.

[0044] Display Modality As used herein, a "clear" image of a surgical scene is an image presented without any special enhancements. Many display modalities exist to enhance the display of surgical images, including, for example, narrow band imaging (NBI), Raman spectroscopy, fluorescence, hyperspectral imaging (HSI), and computed tomography. Each of these enhancement techniques has advantages.

[0045] 11 is an exemplary diagram showing an exemplary clear image of a surgical scene 1100 captured by the camera 528 of the endoscopic display device 101C for display in the viewer 31, according to some embodiments. More specifically, light generated by the light source 531 is reflected from the surgical scene 551 and captured by the camera 528, which generates first information of the image that can be used to configure the computer 58 to generate a clear display image of the surgical scene 551. The tissue region 1102 includes a first anatomical object 1104 and a second anatomical object 1106. In the exemplary surgical scene 1100, the first anatomical object 1104 may include, for example, a prostate or a uterus. The second anatomical object 1106 may include, for example, a nerve, a ureter, or a cervix. The first surgical instrument 1108 and the second surgical instrument 1100 are positioned within the surgical scene 1100. The first surgical instrument 1108 may include, for example, a forceps or a needle driver. The second surgical instrument 1110 may include, for example, scissors or other instruments equipped to deliver monopolar, bipolar or ablative energy.

[0046] FIG. 12 is an exemplary diagram illustrating an example NBI modality image 1200 of the surgical scene 1100 of FIG. 11 in which an image of a vascular anatomical structure 1112 within a first anatomical object 1104 has been enhanced via a narrow-band imaging display modality. Narrow-band imaging (NBI) is an optical imaging technique that improves the visibility of blood vessels 1112 and other structures below the tissue surface. NBI modality images are created through filtered light shining onto the surgical scene. More specifically, ambient light 531 is filtered using a filter 529 to provide specific blue and green wavelengths where hemoglobin peak optical absorption occurs. The reflected light is captured by a camera 528, which generates corresponding second image information including NBI modality image information. The filtered light is shone onto the anatomical tissue to highlight the details of the blood vessels, which appear very dark, allowing for improved visibility and identification of other surface structures. Shorter wavelengths penetrate only the top layers of tissue structures, while longer wavelengths penetrate deeper. Shorter wavelengths are absorbed by capillaries near the surface, making them particularly useful for detecting tumors, which are often highly vascularized. Longer wavelengths are absorbed by blood vessels located deeper within the tissue, which appear different from shallower blood vessels that absorb shorter wavelengths. Longer wavelengths of light that penetrate deeper allow for a better understanding of the vascular structure of suspicious lesions located farther from the tissue surface. In this way, NBI can enhance the visibility of vascular structures within anatomical objects within surgical scenes.

[0047] FIG. 13 is an exemplary diagram illustrating the exemplary Raman spectroscopy modality image surgical scene 1100 of FIG. 11 captured by camera 528, in which an image of a tumor anatomical structure 1114 within a second anatomical object 1106 is highlighted via the Raman spectroscopy display modality. Light reflected from the surgical scene is captured by camera 528. Computer 58 is configured to perform Raman spectroscopy optical image processing of the captured image to generate third (Raman spectroscopy) image information. Raman spectroscopy involves measuring the chemical composition of complex biological samples based on reflected light to generate Raman spectroscopy image information. Certain tumor tissues 1114, for example, have a higher water content than normal anatomical tissues and are therefore distinguishable from surrounding tissues based on their water content. Raman spectra provide quantitative information about the chemical structure of tissues. In particular, Raman spectroscopy is a technique that can be used to observe the inelastic scattering of light by vibrating molecules, providing a chemical fingerprint of cells, tissues, or biological fluids. Raman spectroscopy relies on the scattering of light by molecules, and information about the vibrational modes of molecules can be obtained using a visible or near-infrared laser. A common method for obtaining Raman spectral images is to raster scan a sample through a laser spot or across a sample, then apply a univariate or multivariate spectral model to each Raman spectrum. Thus, Raman spectroscopy can enhance the visibility of anatomical structures within anatomical objects based on their constituent chemical composition.

[0048] Fluorescence is a display modality used to detect fluorescently labeled structures during a surgical procedure. For example, fluorescence can be used to enhance the visual image of fluorescently labeled or unlabeled tissue structures versus non-fluorescent tissue structures. Fluorescence-guided surgery (FGS) can be performed using imaging devices that provide real-time, simultaneous information from color reflectance images and fluorescence emission. Fluorescence modality images are generated by filtering light reflected from the surgical scene. More specifically, one or more light sources 531 are used to excite and illuminate the anatomical tissue region. A camera 528 receives light reflected from the surgical scene that has been filtered with an optical filter 529 that matches the emission spectrum of the fluorophore and generates a fourth image information containing enhanced fluorescence image information.

[0049] FIG. 14 is an exemplary diagram showing an exemplary hyperspectral image 1400 of the surgical scene 1100 of FIG. 11 displayed using a hyperspectral imaging display modality. Hyperspectral imaging (HSI) combines digital imaging and spectroscopy. A hyperspectral camera 533 captures reflected light from the surgical scene, which contains light intensities (radiances) in numerous (typically tens to hundreds) adjacent spectral bands for each pixel in the image. Thus, every pixel in the image contains a continuous spectrum (of emission or reflectance) that can be used to characterize objects in the scene with great precision and detail. More specifically, for each pixel in the image, HSI acquires a three-dimensional data set, called a hypercube, with two spatial dimensions and one spectral dimension. A computer 58 is configured to be used to perform processing to spatially resolve the spectral images obtained by HSI to generate fifth (HSI) image information, providing enhanced visual information regarding tissue physiology, morphology, and composition. Hyperspectral imaging (HSI) can provide a more detailed visual image of a surgical scene than conventional color cameras, which typically capture only three distinct spectral channels corresponding to the visual primaries red, green, and blue. HSI also provides the ability to identify objects or disease states based on the chemical composition of tissue within the sensor's field of view. Thus, hyperspectral imaging offers a significantly improved ability to visually distinguish between objects within a scene based on their spectral characteristics.

[0050] 15 is an exemplary diagram depicting a computed tomography slice display modality of the exemplary surgical scene 1100 of FIG. 11, according to some embodiments. Computed tomography (CT) slices combine images 1500 taken from different angles using a radiation source, typically X-ray radiation, to generate virtual two-dimensional (flat) cross-sectional image slices 1502 of a particular anatomical object, allowing a surgeon to see inside the object without cutting. The cross-sectional images 1502 at different three-dimensional depths within the anatomical object can be displayed separately to provide a visual representation of the internal structures within the object.

[0051] Multi-Modality Display Formats Certain display modalities may be better suited to imaging certain types of anatomical structures than others. For example, anatomical structures such as superficial vasculature or dysplasia can often be more visibly displayed using NBI. However, pathologies such as cancer margins can often be more visibly displayed using, for example, Raman spectroscopy. Furthermore, anatomical structures that can be labeled with fluorescent markers or that can respond to excitation light with their own distinct ("intrinsic") emissions can often be more visibly displayed using, for example, fluorescence. Furthermore, anatomical structures with unique spectral characteristics can often be more visibly displayed using, for example, hyperspectral imaging.

[0052] Additionally, certain viewing modalities may be more suitable for use at certain stages of a surgical procedure than others. More specifically, for example, NBI viewing is often used in the initial planning stages of a head and neck surgical procedure. Raman spectroscopy is often used during post-resection stages of the procedure, but can be used in vivo when combined with Raman-active particles. Fluorescence is often used before and after tissue-removal surgical stages. Hyperspectral imaging is often used during the planning portion of a surgical procedure.

[0053] According to some embodiments, a number of different display formats are provided from which a surgical scene can be selected to display simultaneously using multiple display modalities. The stitching format stitches together video images from two or more different display modalities of a single scene to generate a single surgical scene with different portions represented using the different display modalities. In the stitching format, different image modalities are used to represent different portions of the overall image of the surgical scene, such that only one image modality is visible at any given point within the stitched area of ​​the overall image. The picture-in-picture (PiP) format displays a first, primary, typically full-screen video image of the surgical scene using a first display modality, and displays a reduced-size insert video (or still) image of the surgical scene using a second display modality within a portion of the display. The overlay format overlays an image using one display modality over an image using a different display modality. In the overlay format, different image modalities can be used to represent a single portion of the overall image of the surgical scene, such that the overlay image is partially transparent, allowing what is underneath to be visible.

[0054] Stitch Format FIG. 16 is an exemplary multi-modality display of the surgical scene 1100 of FIG. 11 using multiple display modalities combined using a stitching format in accordance with some embodiments. A first surgical image 1600 of the surgical scene is generated using an NBI modality. A second image 1650 of the surgical scene 1100 is generated using a Raman spectroscopy modality. Note that in the first image 1600, a particular vascular structure 1112 is readily visible, while a tumor structure 1114 within a second anatomical object 1106 is less readily visible. Conversely, in the second image 1650, the tumor structure 1114 is readily visible, while the vascular structure 1112 is less readily visible. A third image 1680 is generated by stitching together a selected portion 1652 of the second display 1650 image and the first display image 1600. It will be appreciated that both images generated using the NBI modality and the Raman spectroscopy modality may be generated during a surgical procedure. According to some embodiments, display rules can determine which portions of one image modality are stitched to a different image modality. In the example of Figure 16, the tumor-containing portion 1652 of the Raman spectroscopy display modality of the second display image 1650 is stitched to the first display image. Thus, in the example of Figure 16, an image of an important anatomical structure that is more easily visible in the Raman spectroscopy modality is stitched to the NBI modality, which is better suited to displaying, for example, vascular structures.

[0055] FIG. 17 is an alternative exemplary multi-modality display of the surgical scene 1100 of FIG. 11 using multiple display modalities combined using a stitching format according to some embodiments. A first image 1600 of the surgical scene is generated using an NBI modality. A second image 1650 of the surgical scene 1100 is generated using a Raman spectroscopy modality. In FIG. 17, a fourth display image 1700 of the surgical scene 1100 is generated by stitching together a selected portion 1602 of the first display image 1600 and the second display image 1650. The NBI display modality portion 1602 of the first display image 1600, including the important vascular image 1112, is stitched to the second display image. Thus, images of important anatomical structures that are more easily visible in the NBI modality are stitched to the Raman spectroscopy modality, which is better suited to displaying tumor structures, for example.

[0056] Picture-in-Picture Format Figure 18 is an exemplary multi-modality display of the surgical scene 1100 of Figure 11 using multiple display modalities combined using a picture-in-picture (PiP) format according to some embodiments. The surgical image of Figure 18 corresponds to the image of Figure 16. However, in Figure 18, a portion 1652 of a second image 1650 generated using a Raman spectroscopy modality is included as a reduced-size PiP image 1800 within the primary first image 1600 generated using an NBI modality.

[0057] Figure 19 is an alternative exemplary multi-modality display of the surgical scene 1100 of Figure 11 using multiple display modalities combined using a picture-in-picture (PiP) format according to some embodiments. The surgical image of Figure 19 corresponds to the image of Figure 17. However, in Figure 19, a portion 1602 of a first image 1600 generated using an NBI spectroscopic modality is included as a reduced-size PiP image 1900 within a primary second image 1650 generated using a Raman spectroscopic modality.

[0058] Annotation Format Figure 20 is an example display of the surgical scene 1100 of Figure 11 using information from multiple display modalities using an annotation format in accordance with some embodiments. The surgical image of Figure 20 corresponds to the images of Figures 16-17. However, in Figure 20, information determined from a portion 2000 of a second image 1650 generated using a Raman spectroscopy modality is included as an annotation 2002 in the primary first image 1600 generated using an NBI modality. The example annotation provides an overlay tumor image 2002 determined from the second image 1650 and a corresponding legend, e.g., "tumor."

[0059] Dynamic Presentation of Multi-Modality Displays Figure 21A is an exemplary flow diagram depicting a first process 2100 for dynamically presenting multi-modality surgical images according to some embodiments. Figure 21B is an exemplary diagram depicting respective first image information 2112 captured using camera 528 and stored in non-transitory storage device 2114 that can be used to generate different image modality images. Figure 21C is an exemplary diagram depicting respective second image portion information 2116 captured using camera 528 and stored in non-transitory storage device 2118 that can be used to generate different image modality image portions. The first process 2100 will be described with reference to surgical system 10.

[0060] The modality selection blocks 2102-1 through 2102-n configure the computer processor 58 to select and generate one or more of the first through nth display modalities. Information used to generate the images can be stored for use in generating a composite multi-modality image displaying multiple image modalities. Exemplary embodiments described herein include NBI, Raman spectroscopy, fluorescence, HSI, OCT, and CT display modalities. It will be appreciated that different image data may be required to generate different display modalities. More specifically, different image data may require the use of different portions of the optical spectrum. For example, NBI modality images generally require specific blue and green wavelengths. Raman spectroscopy images generally require different wavelengths depending on the tissue of interest. Fluorescence images generally require visible and near-infrared wavelengths. HSI generally requires wavelengths throughout the visible and infrared regions. Thus, data for different image modalities may sometimes be captured and processed separately during a surgical procedure. For example, the surgeon may choose to view the fluorescence image first, the NBI image later, and then a composite multi-modality image that includes portions from both the fluorescence and NBI images.

[0061] 21B stores exemplary NBI image information 2120 that can be used to configure the computer 58 to generate a full-screen NBI modality image, such as image 1200 of FIG. 12 , in the viewer 31. The device 2116 stores Raman spectroscopy information 2130 that can be used to configure the computer 58 to generate a full-screen Raman modality image, such as image 1300 of FIG. 13 , in the viewer 31. The device 2116 stores fluorescence information 2140 that can be used to configure the computer 58 to generate a full-screen fluorescence modality image (not shown). The device 2116 stores HSI information 2150 that can be used to configure the computer 58 to generate a full-screen HSI modality image, such as image 1400 of FIG. 14 , in the viewer 31.

[0062] The portion selection blocks 2104-1 through 2104-n configure the computer 58 to select respective portions of one or more of the selected display modalities. Different portions of different image modalities may be selected for display in the composite multi-modality image. For example, a portion of an NBI modality image showing a blood vessel may be selected. A different portion of a Raman spectroscopy modality image showing a tumor may be selected. The storage device 2118 of FIG. 21C stores exemplary NBI image portion information 2122 that can be used to configure the computer 58 to generate an NBI modality image portion, such as image portion 1602 shown in FIG. 17. The storage device 2118 of FIG. 21C stores exemplary Raman spectroscopy image portion information 2132 that can be used to configure the computer 58 to generate a Raman spectroscopy modality image portion, such as image portion 1652 shown in FIG. 16. It will be appreciated that in some embodiments, image portion information 2122 is stored embedded within image portion information 2120 and image portion information 2132 is stored embedded within image information 2130 .

[0063] The format selection block 2106 configures the computer 58 to select a multi-modality display format based on the selected display modalities and the corresponding selected display portions. The block 2108 configures the computer 58 to simultaneously display multiple image modalities of a single surgical scene according to the multi-modality display format selected using the format selection block 2106.

[0064] For example, in some embodiments, modality selection block 2102-1 configures computer 58 to select and generate an NBI image modality, and portion selection block 2104-1 configures computer 58 to determine specific anatomical structures within the NBI image, such as, for example, blood vessels.

[0065] Additionally, for example, in some embodiments, modality selection block 2102-n configures computer 58 to select a Raman spectroscopic image modality, and portion selection block 2104-n configures computer 58 to determine a particular anatomical structure within the Raman image, such as a tumor.

[0066] The Select Other Modalities block (not shown) and Select Other Parts block (not shown) select other display modalities, for example, fluorescence or HSI, and their respective corresponding anatomical structures and parts.

[0067] The format selection block 2106 determines whether a modality will be displayed as primary (e.g., full screen) or as an inset, determines which modalities will be stitched together, which will be displayed as PiP, and determines which modality information will be shown using annotations. The display generation block 2108 configures the computer 58 to display the formatted multi-modality image on a computer display screen.

[0068] FIG. 22 is an exemplary flow diagram detailing a sub-process 2200 for selecting a display modality and a portion of the selected display modality, according to some embodiments. Sub-process 2200 will be described with reference to surgical system 10. In some embodiments, sub-process 2200 implements modality selection blocks 2102-1 through 2102-n and portion selection blocks 2104-1 through 2104-n, shown within dashed line 2110 in FIG. 11 . Decision block 2202 configures computer 58 to determine whether the user has selected a display modality. In response to determining that the user has selected a display modality, block 2204 configures computer 58 to select a portion of the user-selected display modality that includes an image of the anatomical structure of interest. Following selection using block 2204, or in response to decision block 2202 determining that the user has not selected a display modality, control flows to block 2206, which configures computer 58 to receive surgical stage information from surgical system 10. In some embodiments, the surgical stage information includes, for example, surgical system operating status information and surgical instrument kinematic information. Decision block 2208 configures computer system 58 to determine whether the received surgical stage information matches an image selection rule corresponding to the image modality. In response to determining that the received surgical stage information matches the image selection rule, block 2210 configures computer 58 to select a portion of the matching display modality that includes an image of the anatomical structure of interest. In response to a determination by block 2208 that there is no surgical stage match, control returns to decision block 2202.

[0069] 22, it can be seen that different user selections and different surgical stage matches can occur at different points during the performance of a surgical procedure using surgical system 10. Thus, process 2200 can select and generate different image modalities and different portions at different times during the surgical procedure.

[0070] FIG. 23 is an exemplary flow diagram detailing a sub-process 2300 for receiving a user selection of an imaging modality according to some embodiments. The sub-process 2300 is described with reference to the surgical system 10. In some embodiments, the sub-process 2300 implements the decision making block 2202 of FIG. 22. The decision block 2302 configures the computer 58 to determine whether user input is provided to the user control input 36 to select an image display modality. In response to determining that user input is provided to the control input 36 to select an imaging modality, a block 2304 configures the computer 58 to report the user-selected modality to the block 2204. In response to the decision block 2302 determining that user input is not provided to the user control input 36, a decision block 2306 configures the computer 58 to determine whether user eye tracking input is provided to select an imaging modality. In response to determining that user eye tracking input is provided to select an imaging modality, block 2304 configures computer 58 to report the user-selected modality to block 2204. After reporting block 2304 or after decision block 2306, in either case, control flow returns to block 2206.

[0071] 23, it can be appreciated that different user selections can occur at different points during the performance of a surgical procedure using surgical system 10. As can be further appreciated, sub-process 2300 continues to cycle during the surgical procedure, continually checking for changes in user input. Thus, process 2300 can select among different imaging modalities at different times during the surgical procedure.

[0072] FIG. 24 is an exemplary diagram showing an exemplary modality selection user interface UI 2402 displayed within the viewer 31 of the surgical system AA, according to some embodiments. The exemplary modality selection UI display provides modality selections in the left column and receives user input in the right column to indicate the user's modality selection. In some embodiments, the user makes the selection using the control input 36. In the exemplary modality selection, the user selected an NBI modality and a fluorescence modality. Decision block 2302 of FIG. 23 configures the computer to determine what to do when the user makes either modality selection using the modality selection UI display of FIG. 24.

[0073] 25A-25C are exemplary diagrams illustrating two-dimensional eye tracking (FIGS. 25A-25B) and detailed eye tracking (FIG. 25C) according to some embodiments. An eye sensor 47 positioned near where the user looks into the viewer 31 tracks the user's eye movements. With reference to FIG. 25A, a full-screen display has a first modality image 2500, such as a clear modality, displaying an anatomical structure 2502 and having a first stitched-in portion 2504 with a second modality, such as an HSI, centered near a first region 2506 where the user's eyes 2508 are fixating. With reference to FIG. 25B, a full-screen display with a first modality image 2500 displaying an anatomical structure 2502 has a second stitched-in portion 2552 with a second modality centered near a second region 2556 where the user's eyes 2508 are fixating. According to some embodiments, a user gaze lasting at least two seconds is required to trigger the option to generate a stitch-in portion in the area near the user's gaze.

[0074] It will be appreciated that HSI modality images are desirable due to the sharpness of the image and the detailed information contained in the image. However, HSI images typically require significant processing power to generate. Generating HSI modality images for only that portion of the display at which the user gazes can reduce processing requirements and increase overall image processing efficiency. In some embodiments, the control input can be operated to select a planar eye-tracking mode of operation. Decision block 2306 configures the computer to determine what to do when the user makes either planar eye-tracking selection.

[0075] 25C , a screen display shows an anatomical structure 2502 adjacent to a stack of CT depth slices of the anatomical structure shown on the display. A user's gaze is shown focused at a first depth 2582. The user's gaze is also shown focused at a second depth 2584. A first CT slice 2592 corresponding to the first depth 2582 is overlaid on the screen display 2500 showing the anatomical structure 2502 in response to the user's gaze at the first depth 2582. A second CT slice 2594 corresponding to the second depth 2584 is overlaid on the screen display 2500 showing the anatomical structure 2502 in response to the user's second gaze. According to some embodiments, a user gaze lasting at least two seconds is required to trigger the generation of overlaid CT slices at the depth of the user's gaze. In some embodiments, a control input can be actuated to select a depth eye tracking operating mode. Decision block 2306 configures the computer to determine what to do if the user makes any depth eye tracking selections.

[0076] FIG. 26 is an exemplary flow diagram illustrating details of a sub-process 2600 for selecting a multi-modality display format according to some embodiments. The sub-process 2300 is described with reference to the surgical system 10. In some embodiments, the sub-process 2300 implements the format selection block 2106 of FIG. 11. A decision block 2602 configures the computer 58 to determine whether multiple modalities are selected. In response to a determination that multiple modalities are not selected, control returns to the decision block 2602. In response to a determination that multiple modalities are selected, a decision block 2604 determines whether the user specifies a user-selected format. In response to a determination that the user specifies a user-selected format, a block 2606 reports the user-selected format to the block 2108, and control then returns to the decision module 2602. In response to a determination that the user does not specify a user-selected format preference, a block 2608 reports a preset format preference to the block 2108, and control then returns to the decision block 202.

[0077] 26, it can be seen that a user can specify different formats and format priorities at different points during the performance of a surgical procedure using surgical system 10. As can be further seen, sub-process 2600 continues to cycle through the surgical procedure, continually checking for changes in user format selections. Thus, process 2600 can select among different modality formats at different times during the surgical procedure.

[0078] FIG. 27 is an exemplary diagram illustrating an exemplary format selection user interface UI 2702 displayed within the viewer 31 of the surgical system AA, according to some embodiments. The exemplary format selection UI 2702 provides a presentation priority selection 2704 that lists modalities in the left column and receives user input in the middle and right columns to indicate the user's presentation priority selection. In the exemplary priority selection 2706, the user selects both clear and NBI for primary presentation, which in some embodiments is full screen, and selects Raman, fluorescence, and HSI for insert presentation (stitching, PiP, or annotation). The x1 and x2 designations for NBI and clear indicate an exemplary prioritization between NBI and clear, with NBI having a higher priority, respectively. For example, according to the exemplary prioritization, if the NBI modality is present, the NBI modality is primary, but if the NBI is not present, the clear modality is primary. The example format selection UI 2702 also provides a presentation style selection 2706 that lists presentation styles (stitch, PiP, or annotation) in the left column and receives user input in the right column to indicate the user's presentation style selection. In the example style selection 2706, the user selected PiP. Thus, in the example user format selection, NBI is presented as primary, and one or more of Raman, fluorescence, and HSI are presented in PiP within the NBI screen. Decision block 2608 of FIG. 26 configures the computer to use the format selection UI display of FIG. 27 to determine what to do if the user makes either format priority selection.

[0079] FIG. 28 is an exemplary flow diagram detailing a subprocess 2800 for determining whether surgical stage information and corresponding surgical stage rules both indicate a display modality, according to some embodiments. Subprocess 2800 is described with reference to surgical system 10. In some embodiments, subprocess 2800 implements decision block 2208 of FIG. 22. Block 2802 configures computer 58 to receive surgical stage information, such as system operating conditions and surgical instrument kinematics. In some embodiments, the surgical stage information can include additional information, such as patient health record information and surgeon information, such as experience level. Block 2804 configures the computer to receive surgical stage rules for the surgical procedure. Block 2806 configures the computer to apply the received surgical stage rules to the received surgical stage information to determine whether the rules indicate that the received surgical stage information matches the display modality. Block 2808 configures the computer to report the display modality if applicable.

[0080] FIG. 29 is an exemplary diagram depicting an exemplary information structure 2902 including surgical stages and corresponding modalities stored in a storage device within computer system 58, according to some embodiments. Each surgical stage signature corresponds to a respective display modality. According to some embodiments, the occurrence during surgery using a surgical system of surgical stage information that closely matches a surgical stage signature can indicate that the surgical system should use the display modality corresponding to the matching surgical stage vector. According to some embodiments, machine learning techniques can be used to generate surgical stage signatures based on information stored in surgical information atlas 902. More specifically, for example, classifiers can be used in conjunction with expert knowledge to correlate surgical stage signatures with imaging modalities. Each surgical stage signature (SigSS) includes a multidimensional vector. The vector includes vector values ​​indicating attributes of the surgical stage corresponding to the imaging modality. In some embodiments, the surgical system signature is generated based on recorded system state information and recorded surgical instrument kinematic information recorded for multiple surgical procedures using multiple different surgical systems. In some embodiments, for example, the kinematic motion of a surgical instrument is decomposed into multiple vector components representing kinematic features such as instantaneous velocity, instantaneous acceleration, instantaneous three-dimensional position, current motion path, and predicted motion path. Furthermore, in some embodiments, not only is the motion and position of the instrument important in determining a surgical stage signature, but contextual information, such as the physical location of anatomical structures relative to the instrument, the physical location of other instruments, the patient's health, and the nature of the surgery, also plays a role in interpreting the kinematic information. Furthermore, previous instrument motion may be relevant to determining a surgical stage signature, such as which image modality corresponds to a particular surgical stage signature. Thus, in some embodiments, a surgical stage signature may also include vectors representing, for example, the location of anatomical structures, the location of other instruments, the patient's health, the type of surgery, the physician's experience level, and previous instrument motion.

[0081] 28 , block 2806 performs a correlation between the surgical stage signatures, which act as rules, and the received surgical stage information to determine whether there is a sufficient match between the received surgical stage information and the surgical stage signatures to trigger a report to use the display modality corresponding to the received surgical stage information. It will be appreciated that in machine learning embodiments, the match is determined based on the range of similarity between the received instrument kinematics and system state information, etc., and the rules represented by the surgical state signatures. Thus, in some embodiments, for example, surgical stage information within a certain threshold distance from a particular rule is determined to match that rule to trigger selection of the corresponding display modality.

[0082] FIG. 30 is an exemplary flow diagram detailing a sub-process 3000 for generating an image modality portion according to some embodiments. The sub-process 3000 is described with reference to the surgical system 10. In some embodiments, the sub-process 3000 implements blocks 2204 and 2210 of FIG. 22. Block 3002 configures a computer to receive image information captured using the camera 528 for a selected image modality for a procedure performed using the surgical system. The received image information may include, for example, anatomical structure location, margin assessment, and functional information. In some embodiments, the image information may be accompanied by additional information, such as patient health record information and surgeon information, such as experience level. Block 3004 configures a computer to receive an image modality rule. Block 3005 configures a computer system to receive information identifying an image modality corresponding to the received image information and the received rule. Block 3006 configures the computer to apply the received image modality rules to the received image information to identify portions of the image modality that are likely to be of most interest to the surgeon and therefore should be selected for insertion into another image if appropriate to characterize the important portion. Block 3008 configures the computer to report the portion if a corresponding portion is identified.

[0083] It will be appreciated that sub-process 3000 is performed separately for each imaging modality currently being used by surgical system 10 to perform the surgical procedure. More particularly, for example, if only NBI and Raman spectroscopic imaging modalities are currently being used, sub-process 3000 is performed separately for NBI image information and NBI rules and for Raman image information and Raman rules. Furthermore, it will be appreciated that different portions of the overall scene may be identified for different imaging modalities. For example, the NBI modality may be good at identifying blood vessels, while the Raman modality may be good at identifying tumors.

[0084] 31 is an exemplary diagram depicting an exemplary information structure 3102 including image signatures and corresponding annotations stored in a storage device within computer system 58, according to some embodiments. Each image signature corresponds to a respective anatomical structure, such as a blood vessel or tumor, that may be of particular interest to a surgeon during surgery. The annotations can provide a description of the potential relevance of surgical images generated during surgery that closely match the image signature associated with the annotation. Portions of image modality information that match the image signature are selected for insertion (e.g., via stitching, PiP, or annotation) into the overall surgical scene displayed within viewer 31.

[0085] According to some embodiments, machine learning techniques can be used to generate image signatures based on information stored in the surgical information atlas 902. More specifically, for example, a classifier can be used in conjunction with expert knowledge to correlate image signatures with image modalities. Each image signature (SigIM) includes a multidimensional vector. The vector includes values ​​indicative of an anatomical structure. In some embodiments, the image signature is generated based on factors such as an analysis of the shape, spectral signature, depth, and color information captured by the image modality of the anatomical structure. It will be appreciated that different image modalities are suitable for capturing images of different anatomical structures.

[0086] 30 , block 3006 performs a correlation between the image signatures acting as rules and the received image information to determine whether there is a sufficiently close match between the received image information and the image signatures to trigger a report to use the portion of the image corresponding to the received image information. It will be appreciated that in machine learning embodiments, a match is determined based on a range of similarity between the received image information and the rule represented by the image signatures. Thus, for example, in some embodiments, image information within a certain threshold distance of a particular rule is determined to match that rule to trigger the selection of the corresponding portion of the image for display in another image.

[0087] While exemplary embodiments have been shown and described, the foregoing disclosure contemplates a wide range of modifications, variations, and substitutions, and in some instances, some features of the embodiments can be used without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present disclosure is to be limited only by the claims that follow, which claims are appropriately construed broadly in a manner consistent with the scope of the embodiments disclosed herein.

[0088] The following additional note is added: (Supplementary Note 1) A method for generating a multi-modality image of a surgical scene, comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; selecting a portion of the second modality image based at least in part on anatomical structure information contained within the selected portion of the second modality image; configuring a computer to use the first image information and the second image information to simultaneously generate within a display at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene; Including, method. (Supplementary Note 2) The step of selecting the portion of the modality image includes a step of comparing the second image information with image signature information stored in a non-transitory storage device. The method described in Appendix 1. (Supplementary Note 3) The step of selecting the portion of the modality image includes comparing the second image information to a plurality of image signatures stored in a non-transitory storage device. The method described in Appendix 1. (Supplementary Note 4) The configuring step includes configuring the computer to stitch the selected portion of the second modality image into the first modality image in place of the corresponding portion of the first modality image. The method described in Appendix 1. (Supplementary Note 5) The configuring step includes configuring the computer to simultaneously include within the display a reduced-size image of the selected portion of the second modality image and the first modality image. The method described in Appendix 1. (Supplementary Note 6) The first modality image includes a clear image; the second modality image includes an enhanced image; The method described in Appendix 1. (Supplementary Note 7) The first modality image includes an enhanced image; the second modality image includes an enhanced image; The method described in Appendix 1. (Supplementary Note 8) The first modality image includes a clear image; the second modality image comprises a narrowband image; The method described in Appendix 1. (Supplementary Note 9) The first modality image includes a clear image; the second modality image comprises a Raman spectroscopy image; The method described in Appendix 1. (Supplementary Note 10) The first modality image includes a clear image; the second modality image comprises a fluorescence image; The method described in Appendix 1. (Supplementary Note 11) The first modality image includes a clear image; the second modality image comprises a hyperspectral image; The method described in Appendix 1. (Supplementary Note 12) The first modality image includes a clear image; the second modality image comprises an optical coherence tomography image; The method described in Appendix 1. (Supplementary Note 13) Further comprising receiving a format selection at the computer; The configuring step includes configuring the computer to simultaneously generate within the display the at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene in the selected format; The method described in Appendix 1. (Supplementary Note 14) A method for generating a multi-modality image of a surgical scene, comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first enhanced modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second enhanced modality image of the surgical scene based on light captured using the at least one camera; receiving at a computer a format selection designating one of the first and second modality images as a primary and designating the other of the first and second modality images as an insert; selecting a portion of the modality image to be designated as the insert, the portion of the modality image to be designated as the insert being based at least in part on anatomical structure information contained within the selected portion of the modality image designated as the insert; configuring the computer to use the first image information and the second image information to simultaneously generate within a display at least a portion of the modality image designated as the primary and the selected portion of the modality image designated as the insert; Including, method. (Supplementary Note 15) The configuring step includes a step of configuring the selected portion of the modality image designated as the insert to be stitched into the at least a portion of the modality image designated as the primary in place of a corresponding portion of the at least a portion of the modality image designated as the primary. The method described in Appendix 14. (Supplementary Note 16) The configuring step includes a configuring step of simultaneously including within the display a reduced-size image of the selected portion of the modality image designated as the insert and the at least a portion of the modality image designated as the primary. The method described in Appendix 14. (Supplementary Note 17) A method for generating a multi-modality image of a surgical scene, comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; Tracking a user's eye focus position within the display; selecting a portion of the second modality image based at least in part on a focal point of the tracked user's eye within the display; configuring a computer to use the first image information and the second image information to simultaneously generate within the display at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene; Including, method. (Supplementary Note 18) The second modality image includes a hyperspectral image. The method described in Appendix 17. (Supplementary Note 19) The configuring step includes configuring the computer to stitch the selected portion of the second modality image into the first modality image in place of the corresponding portion of the first modality image. The method described in Appendix 17. (Supplementary Note 20) The configuring step includes configuring the computer to simultaneously include within the display a reduced-size image of the selected portion of the second modality image and the first modality image. The method described in Appendix 17. (Supplementary Note 21) A method for generating a multi-modality image of a surgical scene, comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; Tracking the surgical stages during the surgical procedure; selecting a modality image format based at least in part on the tracked surgical stage; configuring a computer to use the first image information and the second image information to generate at least a portion of the first modality image and at least a portion of the second modality image in the selected format within a display; Including, method. (Supplementary Note 22) The step of selecting the modality image includes a step of comparing the surgical stage information with surgical stage signature information stored in a non-transitory storage device. 22. The method described in Appendix 21. (Supplementary Note 23) The step of selecting the modality image includes a step of comparing the surgical stage information with a plurality of surgical stage signature information stored in a non-transitory storage device. 23. The method described in Appendix 22. (Supplementary Note 24) A system for generating multi-modality images of a surgical scene, comprising: processor and; 1. A memory device comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; selecting a portion of the second modality image based at least in part on anatomical structure information contained within the selected portion of the second modality image; configuring a computer to use the first image information and the second image information to simultaneously generate within a display at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene; a memory device holding a set of instructions executable on the processor to cause the system to perform operations including: having system. (Supplementary Note 25) The step of selecting the portion of the modality image includes a step of comparing the second image information with image signature information stored in a non-transitory storage device. 25. The system of claim 24. (Supplementary Note 26) The step of selecting the portion of the modality image includes comparing the second image information with a plurality of image signatures stored in a non-transitory storage device. 25. The system of claim 24. (Supplementary Note 27) The configuring step includes configuring the computer to stitch the selected portion of the second modality image into the first modality image in place of the corresponding portion of the first modality image. 25. The system of claim 24. (Supplementary Note 28) The configuring step includes configuring the computer to simultaneously include within the display a reduced-size image of the selected portion of the second modality image and the first modality image. 25. The system of claim 24. (Supplementary Note 29) The first modality image includes a clear image; the second modality image includes an enhanced image; 25. The system of claim 24. (Supplementary Note 30) The first modality image includes an enhanced image; the second modality image includes an enhanced image; 25. The system of claim 24. (Supplementary Note 31) The first modality image includes a clear image; the second modality image comprises a narrowband image; 25. The system of claim 24. (Supplementary Note 32) The first modality image includes a clear image; the second modality image comprises a Raman spectroscopy image; 25. The system of claim 24. (Supplementary Note 33) The first modality image includes a clear image; the second modality image comprises a fluorescence image; 25. The system of claim 24. (Supplementary Note 34) The first modality image includes a clear image; the second modality image comprises a hyperspectral image; 25. The system of claim 24. (Supplementary Note 35) The first modality image includes a clear image; the second modality image comprises an optical coherence tomography image; 25. The system of claim 24. (Supplementary Note 36) Further comprising receiving a format selection at the computer; The configuring step includes configuring the computer to simultaneously generate within the display the at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene in the selected format; 25. The system of claim 24. (Supplementary Note 37) A system for generating multi-modality images of a surgical scene, comprising: processor and; 1. A memory device comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first enhanced modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second enhanced modality image of the surgical scene based on light captured using the at least one camera; receiving at the computer a format selection designating one of the first and second modality images as a primary and designating the other of the first and second modality images as an insert; selecting a portion of the modality image to be designated as the insert, the portion of the modality image to be designated as the insert being based at least in part on anatomical structure information contained within the selected portion of the modality image designated as the insert; configuring the computer to use the first image information and the second image information to simultaneously generate within a display at least a portion of the modality image designated as the primary and the selected portion of the modality image designated as the insert; a memory device holding a set of instructions executable on the processor to cause the system to perform operations including: having system. (Supplementary Note 38) The configuring step includes a step of configuring the selected portion of the modality image designated as the insert to be stitched into the at least one portion of the modality image designated as the primary in place of a corresponding portion of the at least one portion of the modality image designated as the primary. 38. The system of claim 37. (Supplementary Note 39) The configuring step includes a configuring step of simultaneously including within the display a reduced-size image of the selected portion of the modality image designated as the insert and the at least a portion of the modality image designated as the primary. 38. The system of claim 37. (Supplementary Note 40) A system for generating a multi-modality image of a surgical scene, comprising: processor and; 1. A memory device comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; Tracking a user's eye focus position within the display; selecting a portion of the second modality image based at least in part on a focal point of the tracked user's eye within the display; configuring a computer to use the first image information and the second image information to simultaneously generate within the display at least a portion of the first modality image and the selected portion of the second modality image of the surgical scene; a memory device holding a set of instructions executable on the processor to cause the system to perform operations including: having system. (Supplementary Note 41) The second modality image includes a hyperspectral image. 41. The system of claim 40. (Supplementary Note 42) The configuring step includes configuring the computer to stitch the selected portion of the second modality image into the first modality image in place of the corresponding portion of the first modality image. 41. The system of claim 40. (Supplementary Note 43) The configuring step includes configuring the computer to simultaneously include within the display a reduced-size image of the selected portion of the second modality image and the first modality image. 41. The system of claim 40. (Supplementary Note 44) A system for generating multi-modality images of a surgical scene, comprising: processor and; 1. A memory device comprising: using at least one camera to capture light reflected from the surgical scene; generating first image information in a non-transitory storage device corresponding to a first modality image of the surgical scene based on light captured using the at least one camera; generating second image information in a non-transitory storage device corresponding to a second modality image of the surgical scene based on light captured using the at least one camera; Tracking the surgical stages during the surgical procedure; selecting a modality image format based at least in part on the tracked surgical stage; configuring a computer to use the first image information and the second image information to generate at least a portion of the first modality image and at least a portion of the second modality image in the selected format within a display; a memory device holding a set of instructions executable on the processor to cause the system to perform operations including: having system. (Supplementary Note 45) The step of selecting the modality image includes a step of comparing the surgical stage information with surgical stage signature information stored in a non-transitory storage device. 45. The system of claim 44. (Supplementary Note 46) The step of selecting the modality image includes a step of comparing the surgical stage information with a plurality of surgical stage signature information stored in a non-transitory storage device. 45. The system of claim 44.

Claims

1. One or more processors coupled to a memory: receiving a first modality image of a surgical scene and a second modality image of the surgical scene; receiving surgical stage information associated with the surgical scene; selecting a display modality based on the surgical stage information; simultaneously presenting, on a display, at least a portion of the first modality image of the surgical scene and at least a portion of the second modality image of the surgical scene based on the selected display modality; one or more processors configured to Surgical system.

2. The surgical stage information includes a system operating status; The surgical system of claim 1 .

3. the surgical stage information includes kinematic information of a surgical instrument; The surgical system of claim 1 .

4. the one or more processors are further configured to select the at least a portion of the second modality image based at least in part on anatomical structure information contained within the second modality image. The surgical system of claim 1 .

5. the one or more processors are further configured to select the at least some of the second modality images in response to the surgical stage information matching an image selection rule. The surgical system of claim 1 .

6. the one or more processors are further configured to select the display modality in response to the surgical stage information matching a surgical stage rule. The surgical system of claim 1 .

7. the one or more processors are further configured to use machine learning to generate a surgical stage signature based at least in part on the surgical stage information. The surgical system of claim 1 .

8. the one or more processors are further configured to select the display modality corresponding to the surgical stage signature. The surgical system of claim 7 .

9. One or more processors coupled to a memory: Identifying one or more image signatures for one or more images of one or more surgical scenes during one or more previous surgical procedures; receiving a first modality image of a surgical scene and a second modality image of the surgical scene; identifying a match between an image signature of the one or more image signatures and the first modality image or the second modality image; selecting a portion of the first modality image or a portion of the second modality image based on one or more rules represented by at least the image discrimination characteristics; simultaneously presenting on a display the portion of the first modality image of the surgical scene and the portion of the second modality image of the surgical scene; one or more processors configured to the one or more processors are further configured to receive surgical stage information and select a display modality corresponding to the surgical stage information. Surgical system.

10. the one or more processors are further configured to use machine learning to generate the one or more image signatures based on at least the one or more images. The surgical system of claim 9 .

11. the one or more processors are further configured to use machine learning to identify the match between the image signature of the one or more image signatures and at least one of the first modality image or the second modality image. The surgical system of claim 9 .

12. receiving, by one or more processors of a surgical system, a first modality image of a surgical scene and a second modality image of the surgical scene; receiving, by the one or more processors, surgical stage information associated with the surgical scene; the one or more processors selecting a display modality based on the surgical stage information; presenting, on a display, at least a portion of the first modality image of the surgical scene and at least a portion of the second modality image of the surgical scene based on the selected display modality; A computer-implemented method comprising:

13. The surgical stage information includes system operating state or surgical instrument kinematic information.

13. The computer-implemented method of claim 12.

14. further comprising identifying a match between at least one of the first modality image or the second modality image and one or more image signature characteristics for one or more surgical scenes of one or more previous surgical procedures.

13. The computer-implemented method of claim 12.

15. using machine learning to identify the match between the image signature of the one or more image signatures and the at least one of the first modality image or the second modality image.

15. The computer-implemented method of claim 14.

16. further comprising using machine learning to generate the one or more image signatures.

15. The computer-implemented method of claim 14.

17. selecting at least the portion of the first modality image or the portion of the second modality image based on one or more rules represented by at least the image discrimination characteristics.

15. The computer-implemented method of claim 14.

Images