Three-dimensional medical imaging and interaction

Abstract

A medical image viewer can render to a display a three-dimensional view of a patient's anatomy, a multi-planar reconstruction (MPR) view of the patient's anatomy, and an intraoperative view. Some of these views can be synchronized to show a common focal point of the anatomy. Other embodiments are described and claimed.

Description

[0001] CROSS-REFERENCE

[0002] This patent application claims the benefit of the earlier filing date of U.S. Provisional Application No. 62 / 967,810, filed January 30, 2020, entitled “THREE DIMENSIONAL MEDICAL IMAGING AND INTERACTIONS.” TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of surgical robotics, and more particularly to displaying and interacting with three-dimensional images of a patient using a surgical robotic system. BACKGROUND

[0004] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques aimed at reducing tissue damage during a surgical procedure. For example, a laparoscopic procedure typically involves making a plurality of small incisions in a patient (e.g., in the abdomen), and introducing one or more tools and at least one endoscopic camera into the patient’s body through the incisions. The surgical procedure is then performed using the introduced tools, with visualization assistance provided by the camera.

[0005] Generally, MIS provides multiple benefits, such as reduced patient scarring, reduced patient pain, shortened patient recovery periods, and reduced medical expenses associated with patient recovery. In some embodiments, MIS can be performed using a surgical robotic system that includes one or more robotic arms for manipulating surgical instruments based on commands from an operator. For example, an operator can provide commands for manipulating surgical instruments while viewing images provided by a camera and displayed to a user on a display. However, conventional display systems do not enable effective control of the display system or the surgical robotic system. Moreover, conventional display systems typically provide two-dimensional surgical image data to a user, and current three-dimensional displays typically require users to wear glasses or another similar wearable component (e.g., having polarized filters or dynamic shutters) to visualize three-dimensional images. However, such glasses and other wearable components can present certain issues for use and manipulation in a surgical or sterile environment. During a surgical procedure performed using a surgical robotic system, a surgeon can wish to view image data of a patient that can assist the surgeon in navigating the patient’s anatomy. Accordingly, there is a need for improved three-dimensional display systems that enable users to better visualize a surgical procedure site during a surgical procedure performed using a surgical robotic system. SUMMARY

[0006] Generally, a surgical robotic system having an interactive three-dimensional display can include a left-hand held user interface device (UID) and a right-hand held UID, and an autostereoscopic three-dimensional display. One or more computer processors of the surgical robotic system can be configured to receive a plurality of preoperative images of a patient and perform a reconstruction on the preoperative images to generate a three-dimensional image of the patient. The processors can stereoscopically render the three-dimensional image of the patient, resulting in a stereoscopic data stream having a first set of data representing a left eye position and a second set of data representing a right eye position. The autostereoscopic three-dimensional display can be driven with the stereoscopic data stream to produce one or more views of the three-dimensional image of the patient on the autostereoscopic three-dimensional display. The processors can adjust the view of the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to input from the left UID and / or the right UID. The same UIDs can be used to control a surgical robotic arm (and / or an attached tool) to perform a surgical procedure, allowing a user to seamlessly switch between viewing the three-dimensional image of the patient and performing a surgical procedure on the patient.

[0007] In some embodiments, an endoscopic view of a patient and a surgical procedure site are shown on a three-dimensional display concurrently with a three-dimensional image of the patient. A user can better understand the patient anatomy, which can navigate a surgical tool to affect a surgical procedure. BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1 is a pictorial view of an exemplary surgical robotic system in a surgical setting according to some embodiments.

[0009] FIG. 2 shows a system diagram of a surgical robotic system having a three-dimensional view according to some embodiments.

[0010] FIG. 3 shows a process for providing a three-dimensional view by a surgical robotic system according to some embodiments.

[0011] FIG. 4 shows a user console of a surgical robotic system having a three-dimensional view feature according to some embodiments.

[0012] FIG. 5 shows an exploded side view of an exemplary display or monitor according to some embodiments.

[0013] FIG. 6 and FIG. 7 shows an exemplary display system for use with a surgical robotic system according to some embodiments.

[0014] FIG. 8 and FIG. 9 shows an example of interacting with a three-dimensional image of a patient using a surgical robotic system according to some embodiments.

[0015] FIG. 10 A medical image viewer is shown in accordance with some embodiments.

[0016] FIG. 11 A display system for use with a surgical robotic system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0017] Non-limiting examples of various aspects and variations of the present invention are described herein and shown in the accompanying drawings.

[0018] Referring to FIG. 1 , is a pictorial view of an exemplary surgical robotic system 1 in a surgical setting. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5, such as a table, bed, or the like. The system 1 can incorporate any number of devices, tools, or accessories used to perform a surgical procedure on a patient 6. For example, the system 1 can include one or more surgical tools 7 used to perform a surgical procedure. The surgical tools 7 can be end effectors attached to a distal end of a surgical arm 4 for performing a surgical procedure. In one aspect, the arms 4 can be mounted to a surgical table or bed on which the patient is positioned, as shown in the example of FIG. 1 , or the arms can be mounted to a cart separate from the surgical table or bed.

[0019] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both during a surgical procedure. For example, the surgical tool 7 can be a tool used to access, view, or manipulate the internal anatomy of the patient 6. In one embodiment, the surgical tool 7 is a grasper that can grasp tissue of the patient. The surgical tool 7 can be manually controlled by a bedside operator 8; or it can be robotically controlled via actuated movement of the surgical robotic arm 4 to which it is attached. The robotic arms 4 are shown as a table-mounted system, but in other configurations, the arms 4 can be mounted on a cart, ceiling, or sidewall, or in another suitable structural support.

[0020] In general, a remote operator 9, such as a surgeon or other operator, can use the user console 2 to remotely manipulate the arms 4 or attached surgical tools 7, e.g., teleoperation. The user console 2 can be located in the same operating room as the rest of the system 1, as shown in FIG. 1The user console 2 is shown located in the same room as the patient 6. However, in other environments, the user console 2 can be located in an adjacent or nearby room, or it can be located remotely, e.g., in a different building, city, or country. The user console 2 can include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display, e.g., a view of a surgical site within the patient 6. In the example user console 2, the teleoperator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and handheld UIDs 14 in order to remotely control the arms 4 and surgical tools 7 mounted on the distal ends of the arms 4.

[0021] In some variations, the bedside operator 8 can also operate the system 1 in an “over-bed” mode, where the bedside operator 8 (user) is now located on one side of the patient 6 and simultaneously manipulates both the robotically-driven tools (end effectors attached to the arms 4), e.g., holding a handheld UID 14 in one hand and a manual laparoscopic tool in the other hand. For example, the bedside operator’s left hand can manipulate the handheld UID to control the robotic components, while the bedside operator’s right hand can manipulate the manual laparoscopic tool. Thus, in these variations, the bedside operator 8 can perform both robotically-assisted minimally invasive surgical procedures and manual laparoscopic surgical procedures on the patient 6.

[0022] During an example procedure (surgical operation), the patient 6 is prepared for surgery and draped in a sterile manner with a sterile drape to enable anesthesia. Initial access to the surgical site can be performed manually (to facilitate access to the surgical site) while the arms of the robotic system 1 are in a stowed configuration or a retracted configuration. Once access is complete, initial positioning or preparation of the robotic system 1 including its arms 4 can be performed. The surgical operation then continues with the teleoperator 9 at the user console 2 manipulating the various end effectors and possibly the imaging system with the foot controls 13 and UIDs 14 to perform the surgical operation. Manual assistance can also be provided at the operating bed or table by a draped bedside staff member (e.g., the bedside operator 8), who can perform tasks such as retracting tissue, performing manual repositioning, and tool changes on one or more of the robotic arms 4. There can also be a non-draped staff member to assist the teleoperator 9 at the user console 2. When the procedure or surgical operation is complete, the system 1 and user console 2 can be configured or set into a state to facilitate post-procedure procedures, such as cleaning or sterilization and the entry or printing of health records via the user console 2.

[0023] In one embodiment, the remote operator 9 holds and moves the UID 14 to provide input commands to move the robot arm actuators 17 in the robotic system 1. The UID 14 can be communicatively coupled to the rest of the robotic system 1, e.g., via the console computer system 16. The UID 14 can generate a spatial state signal corresponding to the movement of the UID 14, e.g., the position and orientation of the handheld housing of the UID, and the spatial state signal can be an input signal that controls the motion of the robot arm actuators 17. The robotic system 1 can use control signals derived from the spatial state signal to control the proportional motion of the actuators 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signal and generates corresponding control signals. Based on these control signals that control how the actuators 17 are energized to move segments or links of the arm 4, the movement of the corresponding surgical tool attached to the arm can mimic the movement of the UID 14. Similarly, interactions between the remote operator 9 and the UID 14 can generate, e.g., a clamp control signal that causes the jaws of a grasper of the surgical tool 7 to close and clamp tissue of the patient 6.

[0024] The surgical robotic system 1 can include several UIDs 14, with each UID generating a corresponding control signal for controlling the actuators and surgical tools (end effectors) of the respective arm 4. For example, the remote operator 9 can move a first UID 14 to control the motion of an actuator 17 located in a left robotic arm, with the actuator responding by moving links, gears, etc. in the arm 4. Similarly, movement of a second UID 14 by the remote operator 9 controls the motion of another actuator 17, which in turn moves other links, gears, etc. of the robotic system 1. The robotic system 1 can include a right arm 4 that is fixed to a bed or table on the right side of the patient, and a left arm 4 that is located on the left side of the patient. The actuators 17 can include one or more motors that are controlled such that they drive the articulation of the arm 4 to rotate to change the orientation of an endoscope or grasper of the surgical tool 7 attached to the arm, e.g., relative to the patient. The motion of several actuators 17 in the same arm 4 can be controlled by spatial state signals generated from a particular UID 14. The UIDs 14 can also control the motion of the respective surgical tool graspers. For example, each UID 14 can generate a corresponding clamp signal to control the motion of an actuator (e.g., a linear actuator) that opens or closes the jaws of a grasper at the distal end of the surgical tool 7 to clamp tissue within the patient 6.

[0025] In some aspects, communication between the platform 5 and the user console 2 can be through a control tower 3, which can translate user commands received from the user console 2 (and more specifically from the console computer system 16) into robot control commands that are transmitted to the arms 4 on the robotic platform 5. The control tower 3 can also transmit status and feedback from the platform 5 back to the user console 2. The communication connection between the robotic platform 5, the user console 2, and the control tower 3 can be achieved via wired or wireless links using any suitable data communication protocol of various data communication protocols. Any wired connections can optionally be built into the floor or walls or ceiling of the operating room. The robotic system 1 can provide video output to one or more displays (including displays within the operating room, to remote displays that can be accessed via the internet or other network) (e.g., the robotic system 1 can include one or more endoscope cameras that provide video output or other suitable image data to the displays). The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0026] Reference FIG. 2 FIG. 1 shows a surgical robotic system 1 having features that allow a user (e.g., a surgeon) to browse preoperative images of a patient in a three-dimensional viewing space during a surgery, in accordance with some embodiments. This can provide the surgeon with additional information about the patient’s anatomy that can help the surgeon navigate correctly and perform a surgical procedure on the patient using the surgical robotic system. The same UID and display used to perform the procedure can be used to present the three-dimensional preoperative images, allowing the surgeon to seamlessly view the preoperative images while performing the surgical procedure.

[0027] The surgical robotic system 1 includes multiple UIDs 14, such as a left-hand held UID and a right-hand held UID. In some embodiments, each of the UIDs has a bulb-like portion that is activated when squeezed and deactivated when released. Additionally or alternatively, the UIDs can have one or more buttons that are activated when pressed and deactivated when released. One or more processors 36 are configured to monitor the activation state (e.g., activated or deactivated) of each UID. In some embodiments, each UID can have multiple inputs (e.g., a combination of buttons and / or squeezable bulb-like pieces), each input having an activation state that is monitored by the processor. The processor can track the position and movement of the UIDs, as discussed in other sections.

[0028] The system can use the same hardware used to perform the surgical procedure to seamlessly allow for three-dimensional viewing of pre-operative images during the surgical procedure. For example, the processor is configured to control movement of the plurality of surgical robotic arms 4 and / or tools 7 in response to input from the left UID and / or right UID to perform a surgical procedure on a patient. The user can control the surgical arms on-site or remotely. The display system 140 can be an autostereoscopic three-dimensional display that is used to view real-time images from the operation of the surgical procedure (e.g., endoscope images 44 from an endoscope 42). The autostereoscopic three-dimensional display can simultaneously a) produce a three-dimensional view of the patient and b) produce an endoscope view based on images received from the endoscope. The three-dimensional display can be integrated with the user console so that the same display used to view the three-dimensional images of the patient is also used to view the surgical procedure site within the patient to coordinate movement of the surgical tools in that surgical procedure site. By viewing both views simultaneously, the user can better understand the patient anatomy and how to move the tools to perform the surgical procedure.

[0029] The pre-operative images 43 can be received by the loader and image viewer application 32 from a picture archiving and communication system (PACS) 41. A PACS is a system for storing and accessing medical images from different modalities (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)) that are typically present in hospitals and other medical facilities. The loader and viewer 32 is an application that runs on the surgical robotic system. Prior to a surgical procedure, the system receives pre-operative images of a patient from the PACS. Those images can be stored as multi-frame digital imaging and communications in medicine (DICOM) files. DICOM is a standard that defines how medical images and their metadata are stored and exchanged.

[0030] The system (e.g., at the loader and viewer application 32) can perform a reconstruction on the plurality of pre-operative images 43 to generate a three-dimensional image of the patient. The images, which can be stored as DICOM files, can each represent a“slice” of the patient. These slices can be combined to reconstruct a three-dimensional model of the patient using one or more known techniques that can be performed by the processor. Pre-operative 2D images and reconstructed three-dimensional images can be represented by assigning an intensity to each pixel (2D) or voxel (3D). The system can perform a segmentation, assigning a class to each pixel or voxel of those images in a structure of interest or region of interest (ROI) to produce a segmentation. Such segmentations can be obtained by manual, semi-automatic, or automatic annotation. The three-dimensional visualization, particularly the segmentation, can be used to make structures, such as those that are not currently of interest, transparent. In some embodiments, UID input can select a segmentation of the three-dimensional image of the patient and command those segmentations to become transparent so that the user can better see the region of interest (e.g., an organ or a tumor).

[0031] In addition to this, medical image modalities (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)) can require a reconstruction step after which the acquired data can be viewed in image space. One way to present this data to a user after the reconstruction step is to show three slices or planes—sagittal, coronal, and transverse (also known as axial) slices or planes. Multiplanar reconstruction (MPR) can be performed on preoperative images to generate sagittal, coronal, and transverse views of a patient. MPR involves converting data from an imaging modality acquired in a particular plane, often axial (transverse), to one or more other planes. MPR can be performed using thin slice data from a volumetric CT scan in an axial plane, but other medical image modalities such as MRI and other planes can be used. A medical thin slice image, for example, from an axial plane can then be converted to a non-axial plane, such as a coronal, sagittal, or oblique plane.

[0032] The stereoscopic three-dimensional renderer 34 stereoscopically renders the three-dimensional image of the patient, resulting in a stereoscopic data stream having a first set of data representing a left eye position and a second set of data representing a right eye position. Each set of data (e.g., data stream) represents what each eye of the user sees on the three-dimensional display. In one aspect, the renderer generates two virtual cameras, one for each data stream. The virtual cameras are positioned in a virtual space such that they have a horizontal offset. In one embodiment, off-axis rendering is used to stereoscopically render the three-dimensional image. In this case, each virtual camera has an asymmetric camera frustum. The view frustum is the region of space in the modeled 3D environment that can appear on the screen. In other words, it is the field of view of the virtual camera. The asymmetric camera frustum can avoid viewing strain on the user. The two virtual cameras are placed in the same scene and share a single OpenGL context. Each virtual camera renders the respective data stream to a separate viewport. In contrast, a “toe-in” technique merely aims each virtual camera (without an asymmetric camera frustum) at the same focal point, introducing a vertical parallax that causes viewing strain on the user.

[0033] The display system 140, which utilizes stereoscopic data streams, can be an autostereoscopic three-dimensional display to produce one or more views of the three-dimensional image of the patient on the display. The user can adjust the view of the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to input from the left and / or right UIDs 14. It should be understood that one or more processors 36 execute computer instructions (e.g., stored in non-volatile memory) to run applications such as the load and image viewer application 32, the stereoscopic 3D renderer 34, and other applications. Further, different processors can be dedicated to different tasks, for example, one processor can be dedicated to loading and viewing images. Another processor can be dedicated to stereoscopic rendering. Another processor can be dedicated to monitoring UID activation status. The architecture related to the allocation of processing resources is not relevant to the present disclosure, and such architecture can vary but is not beyond the scope of the present disclosure.

[0034] Referring to FIG. 3 , the process 50 can be performed by a surgical robotic system during a surgical procedure to help provide visual information about the patient anatomy. One or more processors of the surgical robotic system can be configured to perform the process. At block 51, the process includes performing a reconstruction on a plurality of preoperative images of a patient to generate a three-dimensional image of the patient. MPR can also be performed to generate MPR views (e.g., axial, sagittal, and coronal views, as shown in FIG. 7

[0035] At block 52, the process includes stereoscopically rendering the three-dimensional image of the patient, resulting in a stereoscopic data stream having a) a first set of data representing visual data that a left eye would see at a left eye position; and b) a second set of data representing visual data that a right eye would see at a right eye position. As discussed, the virtual cameras can be virtually located in the same scene and share a single OpenGL context to virtually capture images of the three-dimensional image, resulting in corresponding data sets. By “virtual camera,” it is meant that the virtual image data is generated from the point of view of a camera that is simulated by a computer in a computer-simulated space (virtual space).

[0036] ​At block 53, the process includes driving an autostereoscopic three-dimensional display with the stereoscopic data stream to produce one or more views of the three-dimensional image of the patient on the autostereoscopic three-dimensional display. Showing the image to the user in three dimensions requires a three-dimensional monitor. As described, the user console can include a surgical monitor for controlling the surgical robotic arms and attached tools. In some cases, the user can wear 3D glasses while viewing the surgical monitor to produce the three-dimensional effect. In some cases, an autostereoscopic or "glasses-free" display can be deployed in the user console so as to enable it to be unnecessary to wear glasses. The process can generate a visualization pipeline that combines a) the rendered three-dimensional image of the patient and b) the image from the endoscope to form a hybrid view with the three-dimensional image of the patient as well as the real-time feed of the patient surgical procedure that is not three-dimensional.

[0037] At block 54, the process includes adjusting the view of the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to input from a left hand-held user interface device (UID) and a right hand-held UID. The three-dimensional display can be the same three-dimensional display used to control the surgical robotic arms, allowing the user to seamlessly reference the 3D patient anatomy while performing the surgical procedure. Similarly, the UIDs can be the same UIDs used to control the surgical robotic arms. "Clutching" commands such as foot pedal input, voice recognition, or UID input can be used to switch control between using the UIDs to a) control the surgical robotic arms and using the UIDs to b) adjust the view of the patient anatomy on the three-dimensional display.

[0038] The UIDs provide a seamless and intuitive way to interact with the image viewer application. By tracking the position (position and orientation) of each UID, the UIDs can be used as cursors. In some embodiments, the UIDs can have a squeezable portion (e.g., bulbous portion) that, when squeezed, produces an activated state, acting like a button on a computer mouse. Different gestures, movements, and activation sequences can provide different interactions with the user interface, which shows the three-dimensional image and the endoscope view.

[0039] For example, a user can select an image series to be displayed on the UI based on UID input. If the series includes 2D images, the UID can be scrolled (e.g., rotated about a central axis of the UID) to scroll through the set of images. In the case where the series represents three-dimensional image data of a patient (e.g., a series of pre-operative images / slices of a patient), the MPR plane can be changed by rotating the UID and thus scrolling through the slices. Further, gestures with the UID allow for adjusting a reconstructed three-dimensional image of a patient. Zooming is similar to touch screen gestures—pinching two UIDs while bringing the two UIDs closer to each other or further apart. The view can be changed by pinching the two UIDs and simultaneously translating the two UIDs in the same direction. Rotation can be entered by pinching and simultaneously translating only one of the UIDs. A quick pinch and release of the UIDs pops up a menu in which image display options such as brightness and contrast can be selected. Once selected, these parameters can be modified by rotating the UIDs. These interactions are further described in other sections.

[0040] FIG. 4 A schematic view of an example user console 2 is shown. As FIG. 2 A display system 140 for use with the user console 2 and surgical robotic system 1 can be provided, as shown. The display system 140 includes an autostereoscopic three-dimensional display (also referred to as a monitor) 142 that can be driven with autostereoscopic data streams to display three-dimensional (3D) and / or two-dimensional (2D) information to a user. The monitor 142 can display various information associated with a surgical procedure (e.g., three-dimensional images of a patient, endoscopic camera views of a surgical site, static images, GUIs, etc.) or a surgical robotic system (e.g., status, system settings), or other suitable information in the form of 2D and 3D video, image data, text, graphical interfaces, warnings, controls, indicator lights, etc. The monitor 142, as described herein, can also enable a user to interact with displayed content using the user’s eye movements or other suitable gestures in order to control the operation of the display system and other instruments, such as those in a surgical robotic system.

[0041] The three-dimensional display system 140 includes a plurality of sensor assemblies 144 and 146, including a first tracking sensor assembly 144 (which can include a head or eye tracking sensor) and a second tracking sensor assembly 146 (which can include a gaze tracking sensor). The first and second sensor assemblies 144 and 146 can be attached to or, in some variations, integrated with the monitor 142. For example, the first sensor assembly 144 can be connected to an upper or top portion of the monitor 142 and the second sensor assembly 146 can be connected to a lower or bottom portion of the monitor 142, as shown in FIG. 1. In other variations, the first and second sensor assemblies 144 and 146 can be connected to other portions of the monitor 142, such as a side portion or a front portion of the monitor 142. FIG. 2The first sensor assembly 144 and the second sensor assembly 146 are shown attached to the top portion of the monitor 142. However, in an alternative, the second sensor assembly 146 can be attached to the top portion of the monitor 142 and the first sensor assembly 144 can be attached to the bottom portion of the monitor 142, or both the first sensor assembly 144 and the second sensor assembly 146 can be attached to the top portion or the bottom portion of the monitor 142, or the first sensor assembly 144 or the second sensor assembly 146 can be attached to the side portion of the monitor 142. The first sensor assembly 144 or the second sensor assembly 146 can also be coupled to or in conjunction with other suitable components or pieces of the console 2 or in proximity to the console, without departing from the scope of the present disclosure.

[0042] As FIG. 4 As further shown, the monitor 142 can be supported by a power adjustable monitor support assembly 150. The monitor 142 can be positioned proximate or in proximity to the seat 10 to enable a user to view the monitor while the user is seated in or otherwise occupies the seat 10. For example, the support assembly 150 can have a support or column 152 positioned in front of or forward of the seat 10 that at least partially supports the monitor 142. In one variation, the monitor 142 is connected to the support 152 by an adjustable mounting assembly 154 that includes an actuator subsystem having one or more actuators 158, 162, 166 that enable automatic adjustment of the position or orientation of the monitor 152 (e.g., based on output data received from the first sensor assembly 144 or the second sensor assembly 146). The monitor 142 can also include one or more sensors (e.g., position sensors, motion sensors, accelerometers, etc.) attached to the monitor 142 that facilitate detection and tracking of the position or orientation of the monitor.

[0043] The mounting assembly 154 can enable translational or rotational movement of the monitor 142 in up to six degrees of freedom, including, for example, tilt, yaw, roll, forward movement, lateral movement, and vertical movement. For example, the mounting assembly 154 can include a slidable support portion or member 156 coupled to the monitor 142. The slidable support portion 156 can also be driven by one or more actuators 158 (e.g., motors, hydraulic actuators, pneumatic actuators, etc.) to effect vertical and lateral translation of the monitor 142. The mounting assembly 154 can also include one or more telescoping portions or segments 160 or other suitable portions or components that are driven by one or more actuators 162 to effect forward and rearward movement of the monitor 142 (i.e., movement of the monitor 142 toward and away from the seat 10, e.g., to change the distance between the seat 10 and the monitor 142). The telescoping portions 160 can connect the monitor 142 to the slidable support portion 156. The mounting assembly 154 can also include a pivotable connection 164 (e.g., a swivel fixture, a ball joint, a pivot feature, etc.) that connects the monitor 142 to the telescoping portions 160. Movement of the monitor 142 about the pivotable connection 164 can be driven by an actuator 166 (e.g., a motor, a hydraulic actuator, a pneumatic actuator, etc.) to effect tilt, yaw, and roll of the monitor 142. The mounting assembly 154 can also allow for manual adjustment of the position or orientation of the monitor 142.

[0044] The seat 10 can be supported by a power adjustable seat support assembly 165. The power adjustable seat support assembly 165 can have an actuator subsystem including actuators 169 / 171 that drive movement of the seat 10. The seat support assembly 165 includes a seat support 167 having a single strut that at least partially supports the seat 10, although in other examples the seat support 167 can include two or more struts. The seat support 167 can be angled rearwardly relative to the monitor 142, although in other variations it can be straight up vertically angled or tilted forwardly. In some variations, the seat 10 can be movably or adjustably mounted to the seat support 167. For example, the seat 10 can be rotatable, tiltable, reclineable, etc. relative to the support 167 to enable adjustment of the position or orientation of the seat 10 relative to the monitor 142 (e.g., to enable automatic adjustment of the position of a user’s head or eyes relative to the monitor 142 to optimize visualization or perception of three-dimensional imagery on the monitor). The seat assembly 165 can also have one or more actuators 169 (e.g., motors, hydraulic actuators, pneumatic actuators, etc.) for automatically driving rotation, tilting, reclining, etc. of the seat 10 (e.g., in response to output data from the first sensor assembly 144 or the second sensor assembly 146).

[0045] In some variations, the seat 10 can also be movable along the support 167 (e.g., to move the seat 10 up and down and forward and rearward relative to the monitor 142). For example, an actuator 171 (e.g., a motor, a hydraulic actuator, a pneumatic actuator, etc.) can drive the seat 10 to move along the support 167 (e.g., in response to output data from the first sensor assembly 144 or the second sensor assembly 146). Additionally or in the alternative, the seat support 167 can be configured to change its angle or orientation, or to translate in a forward or rearward direction or in a lateral direction. In some further variations, the seat support 167 can be configured to longitudinally or substantially vertically telescope or otherwise extend or retract. The seat support assembly 165 can also allow for manual adjustment of the position or orientation of the seat 10.

[0046] FIG. 5 A cross-sectional view of the system showing the display or monitor 142 is shown. The display or monitor 142 can include a flat, curved, or other shaped panel display 170 (such as an LCD, LED, plasma, or other suitable panel display) having a plurality of pixels for displaying two-dimensional or three-dimensional images. The display 142 can also include one or more layers 172 / 174 at least partially covering or otherwise disposed / positioned over the display 142. The layers 172 / 174 are arranged over a display surface 171 of the display panel to facilitate user visualization of three-dimensional images on the display 142. In one embodiment, the layers 172 / 174 can include lenticules that can be at least partially positioned over the plurality of pixels of the panel display 170 to facilitate or otherwise allow user visualization or perception of three-dimensional images on the panel display 170. In one embodiment, the pixels of the panel display 170 can display left and right eye images that are continuously and / or dynamically alternated, and the layers 172 or 174 can enable the user to visualize or perceive the left and right eye images as a single three-dimensional image without requiring the user to use three-dimensional glasses or other additional wearable or similar components. In the alternative, the one or more layers 172 / 174 can include polarized filters, patterned retarders, or dynamic shutters, and the user can use three-dimensional glasses or other similar wearable components to view or visualize three-dimensional images on the display.

[0047] The display 142 may also include a protective layer 176 that at least partially covers or seals the layers 172 / 174 or the panel display 170. The protective layer 176 seals and protects the layers 172 / 174 and the panel display 170, making the monitor 142 suitable for surgical environments. For example, the protective layer 176 may allow (e.g., using cleaning chemicals, such as alcohol- or chlorine-based cleaners) the display to be sterilized or cleaned without damaging the microlenses 172 / 174. In one embodiment, the protective layer 176 may comprise surgical-grade glass or other surgical-grade materials (e.g., surgical-grade plastic or other suitable composite materials). The protective layer 176 may also have a thickness in the range of about 1 mm to about 3.0 mm (e.g., about 2.0 mm or other suitable integer and non-integer values ​​within this range). However, thicknesses less than 1.5 mm or greater than 3.0 mm may be used without departing from the scope of this disclosure. Alternatively, or in an alternative embodiment, at least one additional protective layer 177 may be provided on the panel display 170 (e.g., between the panel display 170 and layer 170). The additional protective layer 177 may have a thickness of up to 1.0 mm (such as about 0.3 mm) and may be formed of plastic, glass or other suitable materials.

[0048] The protective layer 176 can be bonded to one or both of layers 172 / 174 or panel display 170 using an adhesive 178 (e.g., an optically clear adhesive or other suitable adhesive or glue). One or more spacers 179 may also be provided between the protective layer 176 and layers 172 / 174 or panel display 170. The spacers 179 may be positioned at equal intervals along the boundary of the protective layer 176, but in some variations, the spacers 179 may be provided intermittently or sporadically around the protective layer 176. The spacers 179 prevent damage to layers 174 / 176 during the formation of the monitor (e.g., during the application and bonding of the protective layer 176).

[0049] In some variations, such as FIG. 5 As shown, the first sensor assembly 144 may include one or more sensors 200. Sensors 200 may include a stereo camera, an infrared camera, or other suitable camera 202 that does not filter infrared light. In one embodiment, camera 202 may include one such as those provided by Intel Corp. of Santa Clara, CA. Real-sense camera. Among other things or in the alternative, the sensor 200 can comprise other types of cameras, such as color cameras or other suitable sensing devices, without departing from the scope of the present disclosure. Signals or output information from the first sensor assembly 144 can be received and processed, for example, by a controller or processor in communication with the first sensor assembly 144, to facilitate or otherwise allow detection and tracking of a user's head position or eye position. For example, the first sensor assembly 144 can be used to detect and track a user's head, one eye, or both eyes, for example, an xyz position relative to an origin or original position, such that a position of the user's head or eyes can be continuously determined relative to the monitor 142, for example, a distance.

[0050] Further, in some variations, the second sensor assembly 146 includes one or more sensors 210, such as one or more cameras 212, and one or more flash or flashlights 214 that, for example, flash light to facilitate detection and tracking of a user's gaze by the camera 212. A user's gaze is detected based on a position or movement of at least one iris of the user's eye, and includes an area or point that the user is looking at or substantially focused on (e.g., an area or point on the monitor, or an area or point away from / away from the monitor). In one embodiment, the flash 214 can be configured to provide multiple light flashes per second, for example, at a frequency in a range of about 80 Hz to about 100 Hz, such as about 90 Hz or other suitable frequency. The camera 212 includes a high-speed camera configured to capture the user's iris both illuminated (e.g., by the flash 214) and un-illuminated (e.g., such that a processor receiving and processing output data from the camera 212 can detect and track the user's iris to determine a point or area that the user is looking at or substantially focused on). The light flashes from the flash 214 can also facilitate sensing of a user's eye position or head position with the first sensor assembly 144, for example, in low light conditions.

[0051] It should be appreciated that although some specific examples of sensor types, sensor locations, and sensor functions in a display system have been discussed above, various other sensors and sensor types can be included throughout various components of a display system, in addition to or in the alternative, in order to capture information about a user or to receive user input as interactive user controls.

[0052] FIG. 6An example of a graphical user interface (GUI) to be displayed on monitor 142 is shown. For example, the GUI can display a display portion or window 180 that shows an endoscopic image or other suitable surgical image data (e.g., from an endoscope camera or other suitable surgical robotic camera placed within a patient). The GUI can also include a control panel or side panel 182 that includes one or more images or icons 184 related to one or more applications (e.g., a timer application, an x-ray imaging tool, etc.) that are related to surgical robotic system 1. Control panel 182 can also include other suitable information, such as one or more medical images (e.g., preoperative images of a patient’s tissue), patient data (e.g., name, medical record number, date of birth, various suitable notes, etc.), tool information (e.g., left or right tool number, left or right tool name, left or right tool function, etc.). Other suitable GUIs or other display content can be presented on the monitor without departing from the present disclosure. Various user interactions (e.g., a user’s gaze or eye or head movements) and interactions with the applications as further described below can also result in changes in the type of display content.

[0053] Display system 140 generally includes or is in communication with a processor or controller that is configured to detect and track a user’s head position or eye position relative to monitor 142 based on processing output data of first sensor assembly 144. In some variations, the spatial relationship between monitor 142 and a user (e.g., a user seated in seat 10) can be adjusted based on the detected user’s eye position or head position, for example, to optimize user visualization or perception of three-dimensional image data from an endoscope or surgical robotic camera on monitor 142. Perception of three-dimensional images on monitor 142 by a user can be optimal when the user’s eyes are substantially centered and spaced apart a prescribed distance (e.g., about 70 cm to about 90 cm, such as about 80 cm from the monitor) relative to monitor 142. Accordingly, the position or orientation of seat 10 or monitor 142 can be automatically adjusted or changed (e.g., without requiring intentional user input) to ensure that the user’s head or eyes are positioned in an optimal orientation or viewing distance relative to the monitor.

[0054] In one embodiment, the processor or controller can be in communication with the seat actuators 169 and 171 or the monitor actuators 158, 162, 166 and can automatically provide signals or information to the seat actuators 169 / 171 or the monitor actuators 158, 162, 166 to adjust the position or orientation of the seat 10 or monitor 142 based on processing the output signals from the first sensor assembly 144 or the second sensor assembly 166. For example, the processor or controller can determine the position of the user’s head (or eyes) relative to the monitor 142 and the processor can automatically generate and send a signal to the seat actuators 169 or 171 or the monitor actuators 158, 162, or 166 to adjust or change the position or orientation of the seat or monitor based on the determined position of the user’s head (or eyes) (e.g., the seat can be reclined, tilted, rotated, moved up and down, moved left and right, etc., or the monitor can be tilted, yawed, rotated, moved back and forth, moved left and right, moved up and down, etc.), for example, to optimize the user’s visualization of the three-dimensional images from the surgical robot cameras on the monitor. For example, the position or orientation of the seat or monitor can be adjusted so that the user’s head (or eyes) is substantially centered relative to the monitor and at a prescribed distance from the monitor for optimal viewing of the three-dimensional images.

[0055] Additionally or alternatively, the processor or controller can generate and send a signal to the monitor 142 to display instructions on the monitor for manually adjusting the monitor 142 or seat 10 to optimize the user’s perception or visualization of the three-dimensional image data on the display.

[0056] Further, the controller or processor is configured to detect and track the gaze of the user based on processing the output data of the second sensor assembly 146 and, in some variations, can modify or control the operation of the display system 140 or surgical robot system 1 based on the detected gaze of the user (e.g., to facilitate control of the display system with the user’s eyes or eye pose or to stop or pause the operation of the display system or surgical robot system when the detected gaze of the user is removed from the monitor).

[0057] In some variations, the processor or controller can be in communication with the surgical robot system 1 and can be operable to automatically send a signal or other output data to the surgical robot system 1 or display system 140 to activate or disable one or more operations thereof (e.g., to disable or freeze one or more subsystems of the surgical system, such as the robot arm 4 or surgical tool 7, or to generate an alert with the display system) when the processor or controller determines that the gaze of the user is not directed at the monitor 142 (e.g., for a predetermined period of time, such as about 3 seconds or up to about 5 seconds or more).

[0058] In one embodiment, when the processor or controller determines that the user’s gaze is not directed at the monitor 142, such as when the user is distracted, asleep, etc., for a prescribed period of time, the processor or controller automatically generates and sends one or more signals to the surgical system 1 to freeze or pause operation of the robotic arms 4 or surgical tools 7, for example, to prevent harm to a patient undergoing surgery. Further, when the processor or controller determines that the user’s gaze has returned to the monitor 142, the processor or controller can automatically generate and send one or more signals to the surgical system to resume operation of the robotic arms 4 or surgical tools 7. However, the processor or controller can require a specific user input (e.g., selection of an icon, gesture, etc.) prior to sending the signal to resume operation of the robotic arms or surgical tools 7.

[0059] Additionally or in the alternative, when the processor or controller determines that the user’s gaze is not directed at the monitor 142, the processor or controller can generate and send a signal to the display system 140 to activate one or more alerts or notifications to get the attention of the user or other suitable entity (e.g., a speaker of the display system can play one or more audio sounds, the monitor can display one or more images indicating that the user’s gaze is not directed at the monitor, or one or more vibrations or haptics of the seat or UID can be activated).

[0060] In some variations, the detected and tracked gaze of the user can also be used to initiate or control applications on the control / side panel 182. For example, the user can look at or focus on one or more images 184 on the control or side panel 182 to trigger application interactions. The user can initiate or close an application, open an application in one or more new windows or pop-up windows, control features or operations of an application, etc. by focusing on or looking at one or more areas or points on the GUI or using other suitable eye movements. In one example, the user can focus on or look at an image associated with a timer application shown on the control / side panel to start and stop the timer. In another example, the user can focus on or look at an image associated with an x-ray imaging tool to initiate the x-ray imaging tool (e.g., to open the x-ray imaging tool on one or more secondary or pop-up windows on the display). The gaze of the user can also be used to close the x-ray image tool (e.g., when the user looks away or focuses on a close icon or image or other suitable feature).

[0061] In addition, the position or orientation of the surgical robot camera can be updated or adjusted based on the user's detected and tracked gaze. In some variations, the position or orientation of the surgical robot camera can be updated continuously or dynamically (e.g., a controller or processor can automatically generate signals and send these signals to the actuator subsystem of the surgical robot camera to tilt, rotate, or otherwise translate the lens of the surgical robot camera) such that the area or point on monitor 142 focused by the user is substantially centered along monitor 142 (e.g., centered relative to the horizontal and vertical axes of the monitor), at which point the perception or visualization of the three-dimensional image data is optimal. That is, each time the user focuses on an area or point of the three-dimensional image data displayed in display window 180 that is not substantially centered along display window 180 (e.g., based on the user's detected gaze), the position or orientation of the surgical robot camera can be updated or changed such that the area or point of the three-dimensional image data focused by the user is moved or otherwise adjusted along display window 180 to be substantially centered on the display window.

[0062] For example, a user may initially focus on a point or area of ​​3D image data substantially centered within display window 180, and when the user changes their focus or otherwise redirects their gaze to a new area or point on the image data shown in display window 180 (e.g., the user looks at or focuses on an area or point near or close to the edge or corner of display window 180, or the user looks at or focuses on an area or point otherwise spaced apart from the original point or area in the center of display window), the processor or controller may generate one or more signals and send such signals to the surgical robot camera (or its controller) to automatically adjust the position or orientation of the surgical robot camera such that the new area or point of the 3D image data focused by the user is moved or adjusted to be substantially centered within display window 180. In this way, the position or orientation of the surgical robot camera may be continuously or dynamically adjusted or otherwise updated based on the user's determined gaze, such that the user's focus is on the display window and remains substantially centered along the display window, to facilitate optimal 3D perception or visualization of the 3D image data displayed in the display window.

[0063] During surgery, users can navigate to a view that displays a 3D image of the patient reconstructed from preoperative images, such as... FIG. 7 As shown. Menu items on the GUI can be selected using input from gaze tracking, UID, other inputs (e.g., foot pedal controls), or other known methods (e.g., speech recognition commands). FIG. 6 The navigation is completed using the "patient image" on the screen.

[0064] exist FIG. 7In this case, the image viewer can show all studies and their series associated with the current surgical procedure and patient on a three-dimensional display. While the image viewer is shown, the endoscopic view 256 is still visible in a small window on one side of the screen. A series can be selected for viewing using the UID. In the case where the series contains only 2D images, these images can be displayed, occupying as much of the screen space available to the application as possible. The user can then scroll through the images using the UID, for example, by rotating the images.

[0065] For a series that includes reconstructed data such as CT or MRI, MPR views and 3D renderings are shown. The MPR views can include a sagittal view 248, a coronal view 252, and a transverse (also known as axial) view 242. Through interaction with the UID, the user can scroll through the slices and define those slices to act as clipping planes in the 3D view. Movement in each plane can be described as scrolling through slices of the three-dimensional model of the patient. In some embodiments, input from the UID can be used to select a plane (e.g., axial, sagittal, or coronal plane) relative to the three-dimensional image and scroll through cross-sectional slices of the three-dimensional image of the patient that are parallel to the selected plane.

[0066] For example, the user can generate input using the UID to move the plane 244, which will affect changes in the sagittal view; or move the plane 246, which will affect changes in the coronal view; or move the plane 254, which will affect changes in the axial view.

[0067] The three-dimensional image of the patient 250 is also rendered on the display screen. The user can use the UID to "zoom in," "zoom out," rotate, and move (pan) the reconstructed three-dimensional image around. For example, with reference to FIG. 8 To "zoom in," the system can increase the size of the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to the user simultaneously activating the left and right UIDs 14 and increasing the distance between the left and right UIDs. Conversely, to "zoom out," the system can decrease the size of the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to activating the left and right UIDs and simultaneously decreasing the distance between the left and right UIDs. The amount of size increase or decrease can be proportional to the amount of distance increase or decrease while both UIDs are activated. Thus, in terms of the squeeze-actuated bulb-like UIDs, the user can squeeze the UIDs and pull the UIDs apart to zoom in, and squeeze the UIDs and bring the UIDs together to zoom out.

[0068] In some embodiments, in response to actuation and concurrent shifting of both the left UID and the right UID in the same direction, a three-dimensional image of the patient can be shifted (moved around a three-dimensional space). For example, if the user holds both UIDs away from the body, squeezes and brings the two UIDs closer to the user's body, the three-dimensional image of the patient can be pulled toward the user. If the user holds both UIDs close to the user's body, squeezes and brings the two UIDs away from the user's body, the three-dimensional image of the patient can be pushed away from the user. Other coordinated movements of the user with the UIDs can move the three-dimensional image up, down, left, right, etc. In some embodiments, as shown in FIG. 27, the system can rotate the three-dimensional image of the patient on the autostereoscopic three-dimensional display in response to activation and concurrent rotation of a single UID 14. The UID can be rotated around a longitudinal axis of the UID. FIG. 9

[0069] As mentioned, the system can track the activation state of the UIDs. In addition, the position and orientation of the UIDs are tracked. The UIDs can have trackers built into each UID (inward-outward tracking) or located externally to the UIDs (outward-inward tracking) that track the position (including position and orientation) and movement of the UIDs. In the case of inward-outward tracking, each UID can have a camera (e.g., infrared camera) housed on or in the UID. In some cases, although not required, infrared markers can be placed in fixed positions (e.g., fixed on or around the user console) and tracked by the cameras. Other visual odometry techniques can be used to determine the position and movement of the UIDs based on images captured from the cameras of the UIDs.

[0070] In FIG. 7 response to UID input, a selectable menu 262 can be rendered on the autostereoscopic three-dimensional display. For example, a brief activation period of one of the left or right UIDs can cause the menu to pop up, while another brief activation causes the menu to stow. The menu can have selectable items that adjust display preferences, such as but not limited to contrast, brightness, and / or color. When one of the menu items is selected, the parameters can be adjusted by rotating the UID. The brief activation period can be, for example, less than one second, less than half a second, or less than a quarter of a second.

[0071] ​In some existing medical image viewing solutions, surgeons cannot access patient imaging done for preoperative planning intraoperatively (during surgery). Even in solutions where such imaging is available intraoperatively, other obstacles hinder the usefulness of such information, such as, for example, different image modalities (e.g., CT, MRI, x-ray, ultrasound, etc.), different patient anatomies, and different views of the patient anatomy that are not synchronized and / or shown on different displays. Some existing solutions do not allow for simple creation of annotations and segmentations that can be viewed intraoperatively. Existing solutions are also deficient in that they do not simultaneously show different image studies with different modalities. Further, some solutions that do simultaneously show different images do not perform co-registration of the different images (e.g., from different modalities). Thus, the images are not synchronized to show a common focus, such as an organ, tissue, annotated segmentation, or other focus.

[0072] FIG. 10 A medical image viewer system 310 is shown in accordance with some embodiments. Such a system can be incorporated as a display in a user console of a surgical robotic system (e.g., as shown in FIG. 1 user console 2). The system can include features described in this disclosure, such as those described with respect to FIG. 2 to FIG. 9 the user console 2).

[0073] In FIG. 10 , the system 310 combines preoperative images of a patient (e.g., CT, MRI, and other preoperative imaging) with real-time intraoperative imaging modalities (endoscopic feed, x-ray, ultrasound), allowing for simultaneous visualization of these images. In some embodiments, the system includes fusion (e.g., registration) of multiple modalities of images. The medical image viewer also shows annotations (e.g., surface segmentations) that can be made preoperatively and shown intraoperatively, providing improved surgical guidance.

[0074] The system 310 provides functionality to view preoperative imaging and intraoperative imaging simultaneously in different portions of a display and / or combined in a single view. Images (e.g., of different modalities) can be co-registered, such that direct comparisons and correlations can be made between intraoperative and postoperative views of a patient's anatomy. Images of different modalities can be synchronized in time and space to show a focus of the patient's anatomy.

[0075] At block 300, preoperative imaging of a patient is captured. For example, a CT scanner can generate a CT scan showing the patient's anatomy. Similarly, an MRI can generate MRI images of the patient.

[0076] At block 301, segmentation can be performed using the captured preoperative images. A medical image is represented by assigning an intensity to each pixel (in 2D) or voxel (in 3D) of the image. An image can be divided by assigning a class to each pixel or voxel of those images, resulting in a segmentation of the image. Such segmentation can be obtained by manual annotation and / or semi-automatic annotation. In particular, for 3D visualization, segmentation can be used to make currently uninteresting structures transparent. For example, an image viewer can make anatomical structures that are present in a segmented image but not of interest transparent, improving the visibility of the focus of interest (e.g., a region of interest). The classification for each pixel or voxel can include, for example, organ type (liver, intestine, stomach, etc.), tissue, lesion, annotation, tumor, etc. Thus, each pixel or voxel of the image can have a classification.

[0077] At block 303, the system can receive preoperative annotations from a user. For example, the system can receive input from a user and, based on that input, add a marker or annotation to a landmark, distance, turn, and / or surface shown in the preoperative images taken at block 300 or in the segmented images resulting from block 301. The image viewer can show these annotations intraoperatively. In some embodiments, the system can also call saved viewpoints or perspectives intraoperatively. Thus, annotations that can be made preoperatively, viewpoints created preoperatively, and perspectives created preoperatively can be called to assist during a surgical procedure.

[0078] At block 304, intraoperative imaging is captured. Intraoperative images can include endoscopic images, ultrasound, or other image capture techniques. Intraoperative images are captured in real-time during a surgical procedure. An endoscope can be manually controlled or controlled by a surgical robotic arm, such as those described in other sections.

[0079] At block 308, one or more image registration algorithms are applied to two or more different data sets, such as the pre-operative images from block 300, the segmented images from block 301, the annotated images from block 303, and / or the intra-operative imaging from 304. In some embodiments, the system applies one or more 2D-3D registration algorithms. 2D-3D registration algorithms take a 3D volume, a 2D projection view, and make an initialization or initial guess about where to start the alignment of the 3D volume and the 2D projection. Some algorithms use a trained deep learning model to initialize the optimization procedure and aid the alignment procedure based on different cost metrics such as normalized cross correlation or binary cross-entropy loss. An example of one type of registration algorithm that can be used for 2D-3D registration is an intensity-based deformation registration using mesh-based parameterization as described in Klein, Stefan, et al. “Elastix: a toolbox for intensity-based medical image registration.” IEEE transactions on medical imaging 29.1 (2009): 196-205. Such an algorithm can use B-splines or free-form deformation (FFD) for parameterization. Other registration algorithms can also be applied.

[0080] At block 312, the image viewer renders images of the patient’s anatomy in different views and in different portions of the display 313. In some embodiments, the medical image viewer includes a volume data renderer that displays three-dimensional images from variable angles. In some embodiments, the medical image viewer includes a two-dimensional renderer that displays x-ray or other planar images. In some embodiments, the display can be a stereoscopic display as described in other sections. Additionally or alternatively, the display can be integral with a tablet computer. In some embodiments, the user can generate input with handheld UIDs such as those used to control a surgical robotic system to control the display, select a focal point, select pre-operative annotations, make intra-operative annotations, and / or pan through “slices” of the MPR view. Additionally or alternatively, the user can generate input via a touchscreen user interface. In some embodiments, the focal point can be saved to computer-readable memory and recalled at a later time.

[0081] In some embodiments, the image viewer renders a pre-operative three-dimensional image of the patient’s anatomy to a first portion of the display. At the same time, an MPR view of the anatomy is rendered to a second portion of the display and an intra-operative image of the anatomy is rendered to a third portion of the display. Based on the co-registration of the pre-operative three-dimensional image and the pre-operative image shown in the MPR view, the system can synchronize the three-dimensional image and the pre-operative image to show a common focal point of the anatomy.

[0082] For example, referring back to FIG. 7 , a first portion 281 of the display can show a three-dimensional representation of the patient anatomy 260. The MPR views (242, 248, and 252) can form a second portion of the display. The endoscopic view 256 forms a third portion of the display. The focus of the patient anatomy can be selected to be, for example, an organ (liver, stomach, intestine, etc.), a tumor, or a tissue. The focus can also be any structure or portion thereof of the anatomy that is annotated at block 303. The registration of the images and annotations provides a mapping for the different image modalities and / or annotations so that the images and annotations can be synchronized when viewing a common focus.

[0083] For example, as FIG. 11 shown, if the focus 346 is a tumor, an organ, an annotated structure, etc. in the abdomen, the transparency of other anatomical structures shown in the three-dimensional image (e.g., tissue, bone, muscle, organs) can be increased to highlight the focus. Additionally or alternatively, the focus can be highlighted by adjusting the color and / or brightness of the focus in a manner that contrasts the focus with other anatomical structures. Additionally or alternatively, the three-dimensional image can be arranged (e.g., centered on the focus or rotated image to) emphasize the focus. At the same time, the same focus can be highlighted in each of the MPR views (such as with an outline, color, brightness, or other visual indicator). Additionally or alternatively, the plane of each MPR view (as shown 244, 246, 254) can be arranged to pass through the focus in each MPR view. In this way, the anatomical structures shown can be arranged in different portions of the display and highlighted during a surgical procedure. Different focuses can be selected or invoked to visualize different portions of the anatomical structure from different views. This can help the user navigate surgical tools through the anatomical structure during performance of a surgical procedure. FIG. 7

[0084] The MPR views include at least two of a sagittal view, a coronal view, or a transverse view. In some embodiments, the MPR views include all three views. As mentioned, the MPR views can be arranged based on a focus. The focus can be selected based on input from a user through one or more user interfaces. For example, the user can use a handheld UID as described in other sections to select the focus. The focus can be an item in a menu, or an active selectable portion of any image. In some embodiments, if the display is shown on a touchscreen, the user can provide input through the touchscreen display.

[0085] In some embodiments, the three-dimensional view can be superimposed on and / or aligned with the intraoperative view. For example, referring back to FIG. 7 ​A three-dimensional image of the patient's anatomy 260 can be overlaid on the endoscopic view 256 to provide a three-dimensional understanding of the anatomy shown in the endoscope and to provide additional emphasis on one or more points of interest (e.g., highlighted focal points). The three-dimensional image and / or focal point can be co-registered with the endoscopic feed such that the three-dimensional image is spatially synchronized with the anatomy shown in the endoscopic feed.

[0086] In some implementations, multiple intraoperative imaging sources can be fused at frame 308 and displayed in a fused manner by an image viewer. For example, as FIG. 11 As shown, a first view 340 occupies the third portion of the display, illustrating intraoperative images of patient anatomy captured by a first intraoperative capture device (e.g., an endoscope). The capture device may be located within the patient during surgery to aid in guiding one or more surgical instruments that can be manually or robotically controlled. A second intraoperative view 342 illustrates a second image feed generated using a second capture device 344 (e.g., an ultrasound scanner). The second image feed (ultrasound) may be co-registered with the first intraoperative image feed (endoscope) to create a spatial mapping between the two. This may include, for example, spatial tracking of the endoscope via a position sensor and / or camera. Based on the co-registration of the two image feeds, the system may spatially map the second intraoperative view 342 onto the first intraoperative view 340 (e.g., the endoscopic image) to fuse the second and first views together. In this way, a user can (e.g., using ultrasound) see the “interior” of some anatomical structures that might be invisible by other means (e.g., using a separate endoscope).

[0087] For purposes of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the invention. The foregoing description of specific embodiments of the invention has been provided for illustrative and descriptive purposes. These are not intended to be exhaustive or to limit the invention to the specific forms disclosed; various modifications and alterations can be made to this disclosure in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and its various embodiments with various modifications suitable for the contemplated particular uses.

Claims

1. A surgical robot system, comprising: Left-hand UID and right-hand UID; Automatic stereoscopic 3D display; and Processor, the processor being configured to: Receive multiple preoperative images from the patient; Reconstruction is performed on the multiple preoperative images to generate a three-dimensional image of the patient; The three-dimensional image of the patient is rendered in a stereoscopic manner, thereby generating a stereoscopic data stream; The autostereoscopic 3D display is driven using the stereoscopic data stream to generate one or more views of the 3D image of the patient on the autostereoscopic 3D display; as well as The view of the three-dimensional image of the patient on the automated stereoscopic three-dimensional display is adjusted in response to input from the left-hand handheld UID or the right-hand handheld UID.

2. The surgical robot system of claim 1, wherein the processor is further configured to control the movement of a plurality of surgical robot arms in response to additional input from the left-hand handheld UID or the right-hand handheld UID to perform surgery on the patient.

3. The surgical robot system of claim 1, wherein each UID has a bulb-like element that is activated when squeezed and deactivated when released, and the processor is further configured to monitor whether the bulb-like element is activated or deactivated.

4. The surgical robot system of claim 1, wherein the automated stereoscopic three-dimensional display simultaneously generates an endoscopic view based on images received from the endoscope.

5. The surgical robot system of claim 1, wherein adjusting the view of the three-dimensional image of the patient comprises: Select a plane relative to the three-dimensional image, and scroll through a cross-sectional slice of the three-dimensional image of the patient, the cross-sectional slice being parallel to the selected plane.

6. The surgical robot system of claim 1, wherein adjusting the view of the three-dimensional image of the patient comprises: a) In response to activating the left-handed UID and the right-handed UID and simultaneously decreasing the distance between the left-handed UID and the right-handed UID, the size of the three-dimensional image of the patient on the autostereoscopic three-dimensional display is reduced; or b) In response to activating the left-handed UID and the right-handed UID and simultaneously increasing the distance between the left-handed UID and the right-handed UID, the size of the three-dimensional image of the patient on the autostereoscopic three-dimensional display is increased.

7. The surgical robot system of claim 1, wherein adjusting the view of the three-dimensional image of the patient comprises: The three-dimensional image of the patient shifted within the view in response to activating the left-handed UID and the right-handed UID and simultaneously shifting the left-handed UID and the right-handed UID in the same direction.

8. The surgical robot system of claim 1, wherein adjusting the view of the three-dimensional image of the patient comprises: The three-dimensional image of the patient on the automated stereoscopic display is rotated in response to activation and simultaneous rotation of either the left-handed UID or the right-handed UID.

9. The surgical robot system of claim 1, wherein the processor is further configured to render an optional menu on the automated stereoscopic 3D display in response to a brief period of activation of either the left-handed UID or the right-handed UID.

10. The surgical robot system of claim 1, wherein the automated stereoscopic three-dimensional display comprises one or more layers, the one or more layers being at least partially positioned above the display surface of the automated stereoscopic three-dimensional display to facilitate user visualization of the three-dimensional image on the display surface, the one or more layers comprising at least one of a polarizing filter, a pattern delay unit, or a dynamic shutter.

11. The surgical robot system of claim 10, wherein the user will use glasses or other wearable devices to view the three-dimensional image of the patient on the automated stereoscopic three-dimensional display.

12. The surgical robot system of claim 1, wherein the processor is further configured to: The user's head position relative to the display surface of the automated stereoscopic 3D display is detected and tracked based on data received from an infrared camera; and Modify the operation of the automated stereoscopic 3D display based on the user's detected gaze, and The spatial relationship between the user and the display surface is automatically adjusted based on the user's head position to influence the visualization of the user in the three-dimensional image on the automated stereoscopic three-dimensional display.

13. The surgical robot system according to claim 1, wherein: The three-dimensional image is rendered onto the first part of the automated stereoscopic three-dimensional display. The multi-planar reconstructed MPR view is rendered onto the second part of the automated stereoscopic 3D display. Intraoperative images are rendered onto the third portion of the automated stereoscopic 3D display, and The 3D image and the MPR view are synchronized to show the common focus of the patient's anatomical structures.

14. A method executed by a computing device of a surgical robot system, comprising: Receive multiple preoperative images from the patient; Reconstruction is performed on the multiple preoperative images to generate a three-dimensional image of the patient; The three-dimensional image of the patient is rendered in a stereoscopic manner, thereby generating a stereoscopic data stream; The stereo data stream is used to drive an automated stereoscopic 3D display to generate one or more views of the three-dimensional image of the patient on the automated stereoscopic 3D display; as well as The view of the patient's three-dimensional image on the automated stereoscopic three-dimensional display is adjusted in response to input from a left-handed or right-handed UID.

Images