Hybrid display system for robot-assisted surgery

Abstract

Generating a display to support robotic surgery, comprising: visualising a 3D model of patient anatomy at a first device; controlling the displayed view of the 3D model in response to input from a use

Description

HELD OF THE INVENTION The present application relates to methods and systems for generating a display output to support robotic or laparoscopic surgery. BACKGROUND OF THE INVENTION In robotic and laparoscopic surgery, the surgeon does not have a direct view of the patient’s internal organs. The surgery is done with the aid of a monoscopic or stereoscopic camera, which is either visualized on a two-dimensional (2D) screen (typically for laparoscopic surgery) or a stereoscopic console (typically for robotic surgery). Three-dimensional (3D) laparoscopic screens are now emerging as well, using active or passive stereoscopic glasses. Holographic platforms have been developed for visualizing patient specific 3D holograms of the anatomy and pathology (where a “hologram” in this context refers to any visualized graphical 3D model rather necessarily a hologram in the narrower sense of a 3D image formed by the interference of light), in principle there would likely be benefits in using such holograms intraoperatively (during surgery). However, combining a head mounted display (HMD) with the existing robotic or laparoscopic display devices is either impossible or not practical. For a robotic surgery console, the user would ordinarily have to take the HMD on and off when they need to look at the 3D hologram. For laparoscopic surgery, a high-quality AR headset with camera passthrough (such as AVP), will most likely not give sufficient video quality when the HMD’s passthrough camera is filming a standard laparoscopic 2D screen. Embodiments of the invention thus aim to combine the intraoperative video feed with the 3D model or hologram in a user-friendly way. SUMMARY OF THE INVENTION Aspects of the invention are set out in the independent claims. Certain preferred features are set cut in the dependent claims. Disclosed herein is a method for generating a display output to support robotic surgery performed on a patient, the method comprising: receiving a medical 3D model, the model representative of an aspect of the patient’s anatomy; visualising the 3D model at a display system of a first user device; receiving control input from a user of the first user device to control the displayed view of the 3D model; obtaining image data based on the displayed view of the 3D model, the image data providing a representation of the view shown at the display system of the first user device, the image data being updated responsive to the control input; outputting the image data to a control console for a robotic surgery system, the control console arranged to provide a stereoscopic view during surgery using a stereoscopic display system based on video data received from a stereoscopic camera system; wherein the control console is configured to combine the image data with the stereoscopic view to generate a hybrid display output for output on the stereoscopic display system. The model representative of an aspect of the patient’s anatomy may represent any part(s) or feature(s) of the patient’s body / anatomy, including e.g. bone, organs, other soft tissue, features of patient pathology such as abnormal tissue etc. The console may be pre-configured to combine the image data or may be actively configured (e.g. as a step of the method), for example by sending control data to the console. The control console may be configured to combine the image data with the stereoscopic view using one of: an overlay function, a picture-in-picture display function, or a split-screen display function. The control console may be configured to add an image based on the image data to each frame of a stereoscopic frame pair output via the stereoscopic display system. The image data may comprise a 2D image (e.g. a single such image), wherein the control console adds the (same) 2D image to each frame of a stereoscopic frame pair obtained using the stereoscopic camera system. Alternatively, the image data may comprise a left frame and a right frame of a stereoscopic frame pair, and the control console may add the left and right frames respectively to left and right frames of a stereoscopic frame pair obtained using the stereoscopic camera system of the surgical robot. The first user device preferably comprises a head-mounted display (HMD) device having a stereoscopic display system, wherein the visualising step comprises displaying a stereoscopic view of the model in the stereoscopic display system. The control input to modify the view may be received through an interface of the HMD device. The method may comprise outputting the image data at / from a second user device, the outputting preferably performed via a dedicated video connection, preferably a Digital Video Interface (DVI) connection, to the control console, wherein control input to modify the view is optionally received through the second user device. The second user device preferably receives the medical 3D model and generates a two-dimensional view of the model for display on a display of the second user device in response to control input from a user of the second user device. The 2D image output to the console may correspond to or be based on that two-dimensional view. The views of the 3D model displayed at the first and second user device are preferably synchronized. For example, responsive to control input received through a given one of the first and second user devices, the displayed view of the 3D model at the given device may be modified, and the view shown at the other one of the user devices may be updated according to the control input to correspond to the view at the given device. Preferably, the method comprises performing the generating and outputting steps continuously to generate a video stream of 2D frames or stereoscopic frame pairs to be combined with a stereoscopic video stream at the control console to generate a hybrid video output at the stereoscopic display system, preferably wherein the hybrid view depicted in the hybrid video output changes responsive to the control input from the user at one or both of the first and second user devices. The control input may control one or more of: rotation, zoom, selective display or hiding of parts of the model, selective emphasis or highlighting of parts of the model, positioning or scaling of the model within the hybrid view. The first user device may comprise a stereoscopic display system and / or the first user device may be an Augmented Reality (AR) or Virtual Reality (VR) headset. Also disclosed is a method for generating a display output to support camera-assisted surgery performed on a patient, the method comprising, at a first user device: capturing at least one video stream received from a surgical camera system; transmitting the video stream to a stereoscopic head mounted display (HMD) device; at the HMD device: receiving a medical 3D model, the model representative of an aspect of the patient’s anatomy; generating a stereoscopic view of the model, wherein the stereoscopic view is responsive to control input from a user of the HMD device: combining the stereoscopic view of the model with the received video stream to generate a hybrid view; and displaying the hybrid view in a stereoscopic display system of the HMD device. The surgical camera system is preferably a stereoscopic surgical camera system, the capturing step comprising capturing respective left and right video streams received from the stereoscopic surgical camera system; the method comprising: encoding the left and right video streams into a stereoscopic video stream; and transmitting the stereoscopic video stream to the HMD device, wherein the HMD device combines the stereoscopic view of the model with the stereoscopic video stream to generate the hybrid view. The encoding step preferably comprises encoding respective left and right frames into a single combined frame, preferably with left and right frames placed side-by-side in respective parts of the combined frame, the combined frames forming the stereoscopic video stream, and preferably compressing the stereoscopic video stream for transmission to the HMD device. The left and right video streams are preferably processed by respective video capture devices connected to video outputs of the surgical camera system, the video capture devices connected to or integrated into the first user device. The method may comprise, at the HMD device, recovering left and right frames from the stereoscopic video stream and combining the left and right frames with the stereoscopic view of the 3D model to produce the hybrid view. Preferably, the method comprises generating respective left and right eye projections based on the model and combining the left and right eye projections with respective left and right frames of the stereoscopic video stream, preferably by overlaying, to generate the hybrid view. The method may comprise receiving control input at the HMD device (e.g. from the user of the device via a control interface of the device) to control the view of the 3D model in the stereoscopic display system of the HMD device, wherein the hybrid view is updated in response to the control input. The control input may control one or more of: rotation, zoom, selective display or hiding of parts of the model, selective emphasis or highlighting of parts of the model; positioning or scaling of the model within the hybrid view. In any of the examples described herein, control input may include gesture input, for example using spatial input from one or more HMD controllers, hand / gesture detection or the like, voice input, controller input (e.g. using buttons, sticks / directional inputs), input from a keyboard or pointing device (e.g. mouse) or a combination. The surgical camera system preferably comprises one of: a monoscopic or stereoscopic laparoscopic camera; a stereoscopic surgical camera system of a robotic surgery system. The method may comprise receiving from the robotic surgery system left and right video streams via respective video connections to a control console of the robotic surgery system, wherein the video connections are preferably digital video interface (DVI) connections or other dedicated video connections. The HMD device may be an Augmented Reality (AR) or Virtual Reality (VR) headset. Also disclosed is a system having means (e.g. in the form of one or more processors with associated memory) for performing any method as set out herein. The system may comprise any of the devices or components set out herein, e.g. including one or more of: a user device comprising a stereoscopic head-mounted display (HMD) device; a further user device, e.g. a personal computer; a surgical camera system (e.g. a monoscopic or stereoscopic laparoscopic or surgical robot camera system); a surgical robot and / or control console for a surgical robot. Also disclosed are computer programs or computer-readable media comprising software code configured when executed on a system of one or more user devices (e.g. as set out above) to perform any method as set out herein. Features of one aspect or example may be applied to other aspects or examples, in any combination. For example, method features may be applied to system or computer program aspects or examples (and vice versa). BRIEF DESCRIPTION OF THE FIGURES Certain embodiments of the invention will now be described by way of example only, in relation to the Figures, wherein: Figure 1 illustrates a system for generating a hybrid display output at a robotic surgery console to support robotic surgery, in which a medical 3D model is combined with a video feed from a surgical robot; Figure 2 illustrates a process of generating a hybrid display output in the Figure 1 system; Figure 3 illustrates a system for generating a hybrid display output at an HMD device; Figure 4 illustrates a process for generating the hybrid display output in the Figure 3 system; Figure 5 illustrates processing devices used to implement described processes; and Figures 6A and 6B illustrate hybrid display outputs generated by the described systems. DETAILED DESCRIPTION Embodiments of the invention enable fusion of an intraoperative (usually stereoscopic) video, and a graphical 3D model of a patient’s anatomy and pathology. The described system can be used for robotic surgery (e.g. using a DaVinci robot) or laparoscopic surgery. The fused 3D imaging is visualized in a VR or AR headset such as Apple Vision Pro (AVP) or HoloLens (HL) or at a robotic surgery console. An embodiment of a display system with 2D visualisation of a medical 3D model integrated into a stereoscopic display of the robotic surgery system is shown in Figure 1. The system includes an application suite for medical modelling and surgery planning implemented on a cloud platform 110 (e.g. one or more cloud servers). The application suite includes a medical 3D modelling application 112 for creating medical 3D models which can be visualised in 2D or 3D form and can be used to support planning and execution of surgery. Models may be generated from medical imaging data such as CT (computed tomography) scans. In one approach, CT reconstruction techniques are used to derive a volumetric model from a set of CT scans, where each voxel of the model defines an attenuation value representative of X-ray attenuation at the voxel and is hence indicative of tissue composition at that voxel. Additional processing may be then performed to derive a final medical model from the CT reconstruction volume. For example, segmentation may be performed to label voxels according to tissue type and thus derive a structured medical model. The segmentation may be performed automatically, for example using neural networks or other machine learning approaches, manually using interactive segmentation tools, or using a combination of automatic and manual tools. The resulting segmented 3D model indicates different tissue and / or pathology types (e.g. organ tissue, blood vessels, bone, abnormal tissue etc.) and may be visualised in a 2D projection on a standard computer interface (e.g. using tools to change a displayed view orientation, slice etc). Additionally, the 3D model may be visualised three-dimensionally using suitable 3D display systems such as virtual reality or augmented reality headsets, other passive or active stereoscopic displays, or volumetric displays. Such visualisations may use different colours to label different tissue types according to the performed segmentation, and may allow different tissue types or structures to be displayed or hidden selectively, or emphasised or suppressed to allow a user to inspect the model. In preferred embodiments, medical 3D models may be created using a client device 116, which interacts with the modelling application 112 over a communications network 120 such as the Internet. For example, the modelling application may be implemented as a web application accessed using browser on client device 116, or as a dedicated native application running on the client device. Alternatively or additionally, the user may use a VR / AR headset 118 running a version of the modelling application to create, display and interact with the 3D model. The model may represent a certain part of a patient’s anatomy, e.g. a particular organ, and may be created to assist in planning and carrying out a surgical procedure. To support this, the application suite further includes a surgical planning application 114. This maybe used via devices 116 / 118 to plan surgery e.g. by identifying and marking diseased issue to be removed and planning the required surgical steps to achieve the aims of the surgical procedure. The planning application 114 may use the model created using modelling application 112 as a starting point, and generates a surgery plan, which may e.g. include planning information that is associated with the 3D model, such as regions labelled for removal, locations of incisions etc. in this example, the planning application 114 is depicted as a cloud application forming part of the suite 110 alongside the modelling application 112. In other examples, the planning application may be a dedicated client application running on headset 118 (and / or client device 116). More generally, either of the applications 112, 114 could be provided as cloud applications accessible via web clients running in a browser or via native client applications, as standalone HMD applications for headset 118, standalone PC applications for client device 116, or a combination. In an embodiment, the modelling application 112 and planning application 114 are HoloCare Studio and HoloCare Reality (respectively) available from HoloCare AS, Oslo, Norway. HoloCare Studio is provided as a cloud / web application while HoloCare Reality is an application that is installed and executed on the headset. Note that although these components are described as separate applications they may be part of a single integrated application. During surgery, the surgeon (or surgery team) 150 is supported by a number of systems provided in an operating room environment 140. The principal component is a surgical robot 160, which includes a stereoscopic camera pair 161 for providing left and righteye images to create a stereoscopic view of the patient. The robot is controlled via a surgical robot console 162 including a stereoscopic display 164 (e.g. dual left / right displays viewed via eyepieces integrated into the console). For example, the robot may be a DaVinci (RTM) surgical robot with associated DaVinci console manufactured by Intuitive Surgical, California, USA. Additionally, an AR / VR headset or other HMD device 142 is provided for visualising the 3D model and surgical plan, and a client device 144, e.g. in the form of a standard laptop or desktop personal computer (PC) is provided for interfacing with the cloud platform and / or headset. The client device 144 runs the client components of the application suite 110 (as noted above this may be in the form of native applications or a web browser accessing web applications). The client device 144 is additionally connected to the console 162 via a standard video connection, e.g. a DVI (Digital Visual Interface), HDMI (RTM) (High-Definition Multimedia Interface), or VGA (Video Graphics Array) cable connection or similar. Whilst shown as separate elements, the same HMD device may be used as headsets 118 and 142, for modelling / planning and during surgery, and similarly the client device 144 may be the same as the modeller / planner client device 116. In an embodiment, the headset(s) 118 / 142 comprise Apple Vision Pro (RTM) or Varjo (RTM) headsets. However, other AR / VR systems or head mounted displays (HMD) may be used, such as HoloLens (RTM) headsets manufactured by Microsoft Corporation, Meta Quest (RTM) series headsets etc. The headsets preferably provide high resolution, with high image quality and optical quality to support real-time visualisation during surgery, and are preferably comfortable to wear for long periods. Headsets and client devices may use versions of the modelling and planning client applications adapted to the respective platforms. In the embodiment depicted in Figure 1, a monoscopic (i.e. two-dimensional, non-stereoscopic) version of the 3D model is visualized at the surgical robot console 162 where it is combined with the regular display output of the robotic console on stereoscopic display system 164. A process of generating a hybrid display output using the system is shown in Figure 2. While this description refers to steps being performed by the “user” it will of course be understood that different steps may be performed by different users (for example, different users may perform modelling, planning and / or surgery; during surgery, different users may operate the headset, client device and / or robot console). In step 202, the user creates a medical 3D model using the modelling application 112, e.g. by processing and segmenting a CT reconstruction as described above. In step 204, the user creates the surgical plan using the planning application 114. Modelling and planning may be performed using headset 118, client device 116, or both. Steps 202-204 form the planning phase of the process, whilst the subsequent steps 206-214 form the surgery phase in which the system is used to support ongoing surgery. In step 206, during surgery, the medical 3D model and associated planning information is visualized in headset 142. This can be used by the surgeon when they need to have a full 3D holographic visualization of the patient model. It can also be used by assisting surgeon(s), nurses or students. In step 208, a 2D (monoscopic) view of the model is visualised in a PC version of the planning application (e.g. HoloCare Reality) running on the client device 144. In step 210, the user(s) can interact with the 3D model in the headset and / or the 2D representation on the client device. Interactions can include, for example, rotation, zooming, hiding or showing organ structures or other subdivisions of the model according to the segmentation, emphasising / deemphasising parts of the model (e.g. by changing voxel colours), etc. Headset interaction may use e.g. voice input, gesture input (based on motion-sensitive controllers or hand detection), or controller input (e.g. using control sticks / buttons). Client device interaction may include e.g. keyboard or pointing device / mouse input, touch input, voice commands etc. Any interaction performed in the headset 142 is reflected on the 2D representation shown on the client device 144 and vice versa. As a result, the 2D view on the client device interface may be continuously updated as the user manipulates the view in headset 142, and vice versa, thereby synchronizing the views shown at the two devices. In step 212, the client device encodes the current 2D view shown on the client device interface as an image frame and sends it to the robot console 162 via the video connection. This step is performed continuously during steps 208 / 210, as the user interacts with the headset or client device to update the view. As a result, the robot console receives a continuous 2D video stream corresponding to the 2D visualisation of the 3D model on the client device. In step 214, the surgical console uses split screen functionality, picture-in-picture functionality, overlay functionality or similar functionality of the surgical console, to show the 2D representation received from the client device within the surgeon’s view next to the stereoscopic video feed from the robot. For example, the frames of the 2D video stream may be displayed in a selected corner of each of the left and right images of the stereoscopic image pair so as not to obscure the central region on which the surgeon is focussed. Since the 2D representation is not stereoscopic, the added image portion is the same in each of the stereoscopic images of the robot console display. It will thus appear to the user as a 2D overlay without depth effect within the 3D view created by the stereoscopic display of the robot console. This results in a hybrid view which combines the original stereoscopic video stream from stereoscopic camera system 161 with the video stream of the medical model output by the client device 144. The surgical console may be configured to combine the 2D image from the client device with the stereoscopic feed based on control data sent to the console via a control interface (e.g. via the network). The control data may specify parameters such as the location, size etc for the picture-in-picture / split screen overlay. These parameters may be user configurable. Alternatively, those parameters may be fixed or manually configured at the control console. In some embodiments, instead of an explicit control input, the split-screen function may be activated automatically by the console on detecting a video cable connected to a video port for receiving the 2D image from the client device or upon detecting receipt of a video signal via that connection. A simplified example of the hybrid view is illustrated in Figure 6A, depicting the main stereoscopic view 602, for example showing the surgical tools and tissue being operated on. A picture-in-picture section 604 showing a 2D representation of the 3D model appears in the lower-right corner of each stereoscopic picture (e.g. showing organ structure, blood vessels etc., depending on the segmentation of the model) to create the hybrid view. The main display 602 changes depending on position of the stereoscopic camera and actions performed by the surgeon, whilst the 2D model view 604 changes as the 3D model is manipulated at the client device (144) and / or headset (142). This can allow, for example, a second surgeon operating the headset to highlight aspects of the model to the primary surgeon while the surgeon controls the robot to carry out the surgery. Note that while a separate client device is shown, in other embodiments, the headset could interface directly with the console 162 to output a 2D view based on the current view displayed in the headset. in a variation of the above approach, the robotic console accepts a stereoscopic video stream directly. This could be via a single combined input or via dual video inputs each carrying a separate video stream for the left and right eye. In that case the headset or the client device may output either a single stereoscopic stream or dual streams to the robot console. The robot console then combines left and right frames from the received stream(s), corresponding to the view of the 3D model currently displayed to the user, with the corresponding one of the left and right video streams from the stereoscopic robot in the manner described above, e.g. using split screen, picture-in-picture or an overlay. This produces a fully 3D hybrid view in which both the 3D model and the view from the robot camera system are presented in 3D stereoscopically. As before, the view of the 3D model in that hybrid view responds to user input at the headset and / or client device (e.g. to rotate the model etc.) A second embodiment is depicted in Figure 3. In this approach, the stereoscopic video feed from the robot is fused with the 3D model in the 3D headset for full 3D visualization. Figure 3 shows only the components of the operating room environment; the other system components for use in the modelling / planning phase may be the same as in Figure 1. In this embodiment, the surgical robot console 162 is connected via standard video connections (e.g. DVI, HDMI, VGA etc.) to a pair of 2D capture devices 166, 168. One video connection and capture device are provided for each of the left and right images of the stereoscopic view output generated by the robot console 162 for display on its stereoscopic display. The capture devices 166 and 168 capture the left and right image frames and perform any necessary format conversion and digitization (in the case of an analogue video signal) and provide the captured frames to the client device 144. The capture devices may be industry standard capture cards or devices and are connected to the client device 144 via USB or other suitable connections, or may be integrated into the client device. The client device 144 is in turn connected to the headset 142 (e.g. via a wired or wireless connection) for transmitting the captured frames to the headset. Figure 4 illustrates a process for generating a hybrid 3D view at the headset in accordance with this embodiment. Creation of the model and surgical plan may be performed as previously described with reference to Figure 2 and thus those steps are not repeated here, with Figure 4 showing use of the visualisation functionality during surgery. In step 402, the 3D model is downloaded from the modelling / planning application suite 110 to the headset 142. In step 404, the robot console produces a stereoscopic video feed from the stereoscopic camera 161 inside the patient. In step 406, the left and right signals of the stereoscopic video feed are passed via DVI or other suitable video outputs to a respective video capture device 166 / 168, which capture the left and right video frames and pass the resulting video streams to the client device. In step 408, the client device combines each pair of left and right frames from respective streams received from the capture devices into a single side-by-side frame (with double resolution). This step may typically be performed in software by a converter running on the client device (but alternatively this step could be hardware implemented). This results in a combined stereoscopic video stream where each frame pair is encoded as a single frame. The combined video stream is compressed to H.265 or a similar video compression format. The compression may be performed in hardware / software or a combination. For example, a standard Graphics Processing Unit (GPU) with hardware video encoding may be used (e.g. a GPU manufactured by Nvidia (RTM) or AMD (RTM)). In step 410. the resulting compressed video is streamed frame-by-frame to the headset over a network (e.g. via a local wireless LAN or via a wired connection). In step 412, the headset decodes the compressed video stream (typically in hardware). Each combined frame is split into the left and right frame pairs for visualization on two projection planes in the headset, to enable the user to see the full stereoscopic video. In step 414, a view of the 3D model is generated and overlaid on the frames of the stereoscopic video. Specifically, the headset generates separate 2D projections for the left and right eye displays. The projections are then overlaid on the left and right frames extracted from the received video stream, which are projected onto a rectangle in the virtual HMD space. Preferably, configuration data specifying the field of view and left / right separation of the stereoscopic robot camera are used in generating the hybrid view. For example, eye offset can be set using a GPU shader. The parameters of the stereoscopic camera could be preconfigured for different robot models with the user selecting the appropriate configuration to use, or the user could tune these parameters manually to their liking. Default values could alternatively be used. In step 416, the resulting hybrid view is displayed to the user by outputting the combined left and right frames via the dual left / right eye displays of the headset. As a result, the user sees the stereoscopic video fused with the view of the 3D model. An example is illustrated in Figure 6B, showing the main surgical view 606 derived from the stereoscopic video feed obtained from the robot console, with the superimposed 3D model 608. The user can interact with the model as needed, as described previously (e.g. rotating, zooming, hiding / showing segments etc.), for example using hand gestures or voice commands, while performingthe surgery. The model view in the headset is continuously updated in response to the control input from the user and combined with the frames of the incoming video stream, so that the user will see both the manipulated model and the stereoscopic video stream from the surgical robot within the same view. The hybrid view may display the 3D model overlayed on the stereoscopic video feed at a predefined location (e.g. in one corner) and at a certain size. In some embodiments, the user may control the position and size of the overlaid model (and possibly other parameters such as transparency) to ensure that it does not obscure important details in the robot video feed. The above approaches may also be adapted for use with laparoscopic or other camera-supported surgery systems. For laparoscopic surgery, the technical implementation will generally be the same as for robotic surgery, except that the stereoscopic video stream is received from a stereoscopic laparoscopic camera such as a B.Braun EinsteinVision (RTM) instead of a robotic console. In some embodiments, the video may be a monoscopic video stream. In this case, the 3D model is fused with a traditional 2D video to create a hybrid 3D / 2D view in the headset. Figure 5 illustrates processing devices suitable for implementing the client device(s) 116 / 144 and headset(s) 118 / 142 of the system. The client device 116 / 144 may be based on conventional personal computer hardware and as such includes one or more processors 508 together with a main memory 502 (e.g. volatile / random access memory) for storing temporary data and software code being executed. An input / output subsystem 506 includes one or more I / O interfaces for communicating with external devices and peripherals, such as displays, input devices (e.g. keyboard, mouse), external storage devices and the like. A network interface 510 is provided for communication with external systems via the network 120 (encompassing e.g. Local and / or Wide Area Networks, including private networks and / or public networks such as the Internet, cellular telephony networks etc.) For example, the client device may communicate with the headset 118 / 142 and the application platform 110 (Figure 1) via the network. The client device may further include a GPU 514 for generating 2D views of the 3D model and for encoding / decoding image / video data, and a pair of capture cards 516 / 518 for capturing video output from the robot console (in the Figure 3 embodiment). Persistent storage 504 (e.g. in the form of hard disk storage, optical storage and the like) persistently stores software and data for performing various described functions, such as the client application(s) and / or browser application used for interacting with the application suite 110, video decoding / encoding functions etc. The persistent storage further includes a computer operating system and any other software and data needed for operating the processing device. The device may include other conventional hardware components as known to those skilled in the art. The various components are interconnected by one or more data buses 512 (e.g. system / memory bus and one or more I / O buses). The headset 118 / 142 includes a network interface 526 for communicating with network 120 and client device 116 / 144 (as well as the application platform 110). A processing subsystem 524 (e.g. mobile device processor, memory, and persistent storage) runs an operating system of the headset and any applications, including a client application for the application suite 110 which can download, display, and allow interaction with 3D models and surgical plans. The headset may also include a hardware image / video decoder 522 for decoding stereoscopic images and video streams as discussed above. A display interface 530 provides access from the processing subsystem to a pair of displays 530, 532 for displaying stereoscopic frame pairs to the left and right eyes of the user, thereby enabling stereoscopic display of 3D content. The components are interconnected via bus 528. The headset may further include other standard hardware / software components. While a specific architecture is shown and described by way of example, any appropriate hardware / software architecture may be employed to implement the various components. Furthermore, functional components indicated as separate may be combined and vice versa. Functions may generally be implemented in hardware, software ora combination. It will be understood that the present invention has been described above purely byway 5 of example, and modification of detail can be made within the scope of the invention.

Claims

1. A method for generating a display output to support robotic surgery performed on a patient, the method comprising:receiving a medical 3D model, the model representative of an aspect of the patient’s anatomy:visualising the 3D model at a display system of a first user device;receiving control input from a user of the first user device to control the displayed view of the 3D model;obtaining image data based on the displayed view of the 3D model, the image data providing a representation of the view shown at the display system of the first user device, the image data being updated responsive to the control input; andoutputting the image data to a control console for a robotic surgery system, the control console arranged to provide a stereoscopic view during surgery using a stereoscopic display system based on video data received from a stereoscopic camera system;wherein the control console is configured to combine the image data with the stereoscopic view to generate a hybrid display output for output on the stereoscopic display system.

2. A method according to claim 1, wherein the control console combines the image data image with the stereoscopic view using one of: an overlay function, a picture-in-picture display function, ora split-screen display function.

3. A method according to claim 1 or 2, wherein the control console adds an image based on the image data to each frame of a stereoscopic frame pair output via the stereoscopic display system.

4. A method according to any of the preceding claims, wherein the image data comprises a 2D image, wherein the control console adds the 2D image to each frame of a stereoscopic frame pair obtained using the stereoscopic camera system.

5. A method according to any of claims 1 to 3, wherein the image data comprises a left frame and a right frame of a stereoscopic frame pair, and wherein the control console adds the left and right frames respectively to left and right frames of a stereoscopic frame pair obtained using the stereoscopic camera system.

6. A method according to any of the preceding claims, wherein the first user device comprises a head-mounted display (HMD) device having a stereoscopic display system, wherein the visualising step comprises displaying a stereoscopic view of the model in the stereoscopic display system.

7. A method according to claim 6, wherein the control input to modify the view is received through an interface of the HMD device.

8. A method according to any of the preceding claims, comprising outputting the image data at a second user device to the control console, the outputting preferably performed via a dedicated video connection, wherein control input to modify the view is optionally received through the second user device.

9. A method according to claim 8, wherein the second user device receives the medical 3D model and generates a two-dimensional view of the model for display on a display of the second user device in response to control input from a user of the second user device.10, A method of claim 8 or 9, comprising synchronizing the views of the 3D model displayed at the first and second user devices.

11. A method according to any of claims 8 to 10, comprising, responsive to control input received through a given one of the first and second user devices, modifying the displayed view of the 3D model at the given device, and updating the view shown at the other one of the user devices according to the control input to correspond to the view at the given device.

12. A method according to any of the preceding claims, comprising performing the generating and outputting steps continuously to generate a video stream of 2D frames or stereoscopic frame pairs to be combined with a stereoscopic video stream at the control console to generate a hybrid video output at the stereoscopic display system, preferably wherein the hybrid view depicted in the hybrid video output changes responsive to the control input from the user at one or both of the first and second user devices.

13. A method according any of the preceding claims, wherein the control input controls one or more of: rotation, zoom, selective display or hiding of parts of the model, selective emphasis or highlighting of parts of the model, positioning or scaling of the model within the hybrid view.

14. A method according to any of the preceding claims, wherein the first user device comprises a stereoscopic display system, optionally wherein the first user device is an Augmented Reality (AR) or Virtual Reality (VR) headset.

15. A system having means for performing a method according to any of the preceding claims, the system optionally comprising one or more of:a user device comprising a stereoscopic head-mounted display (HMD) device;a further user device;a surgical camera system;a surgical robot and / or control console for a surgical robot.

16. One or more computer-readable media comprising software code configured when executed on a system of one or more user devices to perform a method as set out in any of claims 1 to 14.s

Images