Spatially aware display for computer-assisted intervention

Abstract

Systems, methods, and techniques for spatially-aware displays for computer-aided interventions are described herein. A fixed frustum-of-visibility technique renders computer images on a display using a perspective based on a virtual camera having a field of view facing the display, and automatically updates a virtual position of the virtual camera in response to adjusting a pose of the display. A dynamic mirror frustum-of-visibility technique renders computer images on a display using a perspective based on a field of view of a virtual camera having a virtual position located behind the display device. The virtual position of the virtual camera is dynamically updated in response to movement of a viewpoint of a user located in front of the display device. Slice visualization techniques for use with the fixed frustum-of-visibility technique and the dynamic mirror frustum-of-visibility technique are also described herein.

Description

[0001] Cross-references to related applications

[0002] This international patent application claims priority to U.S. Provisional Patent Application No. 63 / 036,559, filed June 9, 2020, the entire contents of which are incorporated herein by reference. Background Technology

[0003] In the very early days of X-ray fluoroscopy, radiologists used a fluorescent handheld screen, which was placed in the beam of light emitted from the X-ray source and passed through the patient. The image was directly related to the X-ray source, patient, screen, and observer because it was viewed directly from the location where the image was produced. With advancements in imaging and display technologies, static or real-time images of the patient's anatomy can be viewed from any location where the monitor can be positioned. This has the advantages of more practical patient and monitor positioning and reduced doses experienced by interventional radiologists, but it loses the intuitive perception of the spatial configuration between the X-ray source, patient, screen, and viewer.

[0004] Essentially 3D CT or MRI scans are typically visualized as a set of three orthogonal slices along the patient's anatomical axis or along the instrument's axis. Until recently, 3D images were often rendered on a monitor in the operating room in addition to slice visualization. These are typically volumetric renderings from a viewpoint, which can be defined and controlled by the user. Many surgeons prefer 2D images to 3D graphic renderings because they are well-trained in interpreting 2D images, and this has thus defined the current clinical standard.

[0005] Navigation in surgery has been introduced since the early 1990s and has utilized various visualization techniques. Many systems display slices with additional annotations of preoperative or intraoperative data. These annotations relate to the surgical plan that needs to be implemented during the procedure. Such annotations are typically attempts to visualize deep anatomical targets and the surgeon's safe path to them based on preoperative images. In such systems, the surgeon's position and the monitor's position are not considered when rendering the images. Therefore, the monitor's position does not affect the visualization, nor does the observer's position.

[0006] A series of related works are often referred to as augmented reality windows (AR windows). Existing techniques track translucent displays used for in-situ medical augmentation, which combine head trackers and stereoscopic glasses. Such translucent displays comprise a semi-silvered glass panel that reflects an image from a computer monitor. Other techniques have addressed the same problem of creating AR windows on a patient's anatomy using translucent displays between the patient and surgeon. However, these existing techniques replace the semi-silvered glass panel with an active-matrix LCD. Later attempts again utilized a semi-silvered glass panel to address the problem, which prevents projection from two DLP projectors that generate high-contrast images. Other systems consist of a tracked, moving, opaque screen, where the screen's position affects visualization. Again, such existing systems are placed between the surgeon and patient, displaying sliced ​​views of the anatomy. However, in this system, the user's perspective is not considered; the image on the screen is only two-dimensional, independent of the surgeon's viewpoint.

[0007] Other techniques have proposed AR visualizations inspired by methods dentists use to examine a patient's mouth without changing their viewpoint. These techniques acknowledge that rotating or moving an object around it is not possible in some AR applications. It is proposed to generate additional virtual mirror views to provide an auxiliary perspective on virtual objects within the AR view. Spatially tracked joysticks are used to move virtual mirrors reflecting virtual data like real mirrors within the AR view of a head-mounted display (HMD). Summary of the Invention

[0008] The present invention is presented in a simplified form, with the following conceptual choices further described in the detailed implementations. This invention is not intended to limit the scope of the claimed subject matter and does not necessarily identify each and every key or essential feature of the claimed subject matter.

[0009] In a first aspect, a system for facilitating interaction with a physical object is provided, the system comprising: a display device defining a plane and located on a first side of the physical object; and a navigation system coupled to a control system and configured to: register a computer image with the physical object; track the pose of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer image according to the tracked pose of the physical object and the display device; wherein the rendering perspective is based on a virtual camera having a virtual position located on a second side of the physical object opposite to the first side, and wherein the virtual camera has a field of view facing the plane of the display device, and wherein the virtual position of the virtual camera is automatically updated in response to adjustments to the pose of the display device.

[0010] In a second aspect, there is provided a navigation system of the system of the first aspect.

[0011] In a third aspect, there is provided a method of operating a system for assisting in interacting with a physical object, the system comprising: a display device defining a plane and being located on a first side of the physical object; and a navigation system coupled to a control system, the method comprising: registering a computer image with the physical object; tracking poses of the physical object and the display device in a common coordinate system; and controlling the display device to render the registered computer image in accordance with the tracked poses of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera having a virtual position located on a second side of the physical object opposite the first side, and wherein the virtual camera has a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in response to adjusting the pose of the display device.

[0012] In a fourth aspect, there is provided a computer program product for assisting in interacting with a physical object, the computer program product being usable with a system comprising: a display device defining a plane and being located on a first side of the physical object; and a navigation system coupled to a control system, the computer program product comprising instructions configured to, when executed by one or more processors: register a computer image with the physical object; track poses of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer image in accordance with the tracked poses of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera having a virtual position located on a second side of the physical object opposite the first side, and wherein the virtual camera has a field of view facing the plane of the display device, and wherein the virtual position of the virtual camera is automatically updated in response to adjusting the pose of the display device.

[0013] In a fifth aspect, a system for assisting in interacting with a physical object is provided, the system comprising: a display device defining a plane, wherein the physical object is located in front of the plane; a navigation system coupled to a control system and configured to: register a computer image with the physical object; track poses of the physical object, the display device, and a viewpoint of a user in a common coordinate system; and control the display device for rendering the registered computer image in accordance with the tracked poses of the physical object, the display device, and the viewpoint of the user; and wherein a perspective of the rendering is based on a field of view of a virtual camera having a virtual position located behind the plane, and wherein the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the viewpoint of the user.

[0014] In a sixth aspect, the navigation system of the system of the fifth aspect is provided.

[0015] In a seventh aspect, a method of operating a system for assisting in interacting with a physical object is provided, the system comprising: a display device defining a plane, wherein the physical object is located in front of the plane; and a navigation system coupled to a control system, the method comprising: registering a computer image with the physical object; tracking poses of the physical object, the display device, and a viewpoint of a user in a common coordinate system; and controlling the display device for rendering the registered computer image in accordance with the tracked poses of the physical object, the display device, and the viewpoint of the user; and wherein a perspective of the rendering is based on a field of view of a virtual camera having a virtual position located behind the plane, and the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the viewpoint of the user.

[0016] In an eighth aspect, a computer program product for assisting in interacting with a physical object is provided, the computer program product usable with a system comprising: a display device defining a plane, wherein the physical object is located in front of the plane; and a navigation system coupled to a control system, the computer program product comprising instructions configured to, when executed by one or more processors: register a computer image with the physical object; track poses of the physical object, the display device, and a viewpoint of a user in a common coordinate system; and control the display device for rendering the registered computer image in accordance with the tracked poses of the physical object, the display device, and the viewpoint of the user; and wherein a perspective of the rendering is based on a field of view of a virtual camera having a virtual position located behind the plane, and wherein the virtual position of the virtual camera is automatically updated in response to movement of the tracked pose of the viewpoint of the user.

[0017] In a ninth aspect, there is provided a system, method or computer program product for assisting in interacting with a physical object, comprising: a display device defining a plane; and a navigation system coupled to a control system and configured to: register a computer image with the physical object; track poses of the physical object and the display device in a common coordinate system; and control the display device to render the registered computer image in accordance with the tracked poses of the physical object and the display device; and wherein a perspective of the rendering is based on a virtual camera having a field of view facing the plane of the display device, and wherein the virtual position of the virtual camera is automatically updated in response to adjustments to the pose of the display device.

[0018] In a tenth aspect, there is provided a system, method or computer program product for assisting in interacting with a physical object, comprising: a display device defining a plane; a navigation system coupled to a control system and configured to: register a computer image with the physical object; track poses of the physical object, the display device and a viewpoint of a user in a common coordinate system; and control the display device for rendering the registered computer image in accordance with the tracked poses of the physical object, the display device and the viewpoint of the user; and wherein a perspective of the rendering is based on a field of view of a virtual camera, wherein the virtual position of the virtual camera is automatically updated in response to movements of the tracked pose of the viewpoint of the user.

[0019] Any of the above aspects can be combined, in part or in whole.

[0020] Any of the above aspects can be combined, in part or in whole, with any of the following implementations, either individually or in combination:

[0021] In one implementation, the computer image of the physical object originates from a 3D model. In one implementation, the control system is configured to control the display device to display one or more slices of the 3D model. In one implementation, the slices are displayed in accordance with the tracked poses of the physical object and / or the display device. In one implementation, the one or more slices are sliced at a plane that is aligned with the plane of the display device. In one implementation, the plane is parallel to the display device. In one implementation, the control system is configured to control the display device to automatically change the one or more slices to other slices in response to at least one or more of: the tracked pose of the display device; the tracked pose of the physical object; the tracked pose of a surgical instrument; and / or the tracked pose of a viewpoint of a user.

[0022] In one implementation, the virtual position of the virtual camera is at a predetermined distance from a plane of the display device. In one implementation, the predetermined distance is automatically updated. In one implementation, the predetermined distance is automatically updated in response to at least one or more of: a tracked pose of the display device, a tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user’s viewpoint; a type of surgical procedure; and / or a particular step of a surgical procedure. In one implementation, the predetermined distance is manually updated in response to user input. In one implementation, the predetermined distance is fixed.

[0023] In one implementation, the virtual position of the virtual camera is automatically updated relative to an X-Y arrangement of the plane of the display device. In one implementation, the X-Y arrangement is automatically updated. In one implementation, the X-Y arrangement is automatically updated in response to at least one or more of: a tracked pose of the display device, a tracked pose of the physical object, a tracked pose of a surgical tool; a tracked pose of a user’s viewpoint; a type of surgical procedure; and a particular step of a surgical procedure. In one implementation, the virtual position of the virtual camera relative to the X-Y arrangement of the plane of the display device is manually adjustable based on user input. In one implementation, the virtual position of the virtual camera relative to the X-Y arrangement of the plane of the display device is fixed.

[0024] In one implementation, the pose of the display device is manually adjustable. In one implementation, the virtual position of the virtual camera is automatically updated in response to manual adjustment of the pose of the display device. In one implementation, one or more actuators are coupled to the display device. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device. In one implementation, an input device is coupled to the control system, and the control system is configured to receive commands from the input device and control the one or more actuators to adjust the pose of the display device according to the commands. In one implementation, a surgical instrument includes one or more trackable features, and the navigation system is configured to track a pose of the surgical instrument in the common coordinate system and control the display device to display an image of the surgical instrument. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument. In one implementation, a viewpoint tracking system is coupled to the navigation system and configured to track a pose of a viewpoint of a user in the common coordinate system. In one implementation, the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the viewpoint of the user.

[0025] In one implementation, a line is defined between the virtual position of the virtual camera and the plane of the display device. In one implementation, the line is transverse to the plane of the display device. In one implementation, the line is orthogonal to the plane of the display device. In one implementation, the line falls at a geometric center of the display device. In one implementation, the line falls at a location offset from a geometric center of the display device.

[0026] In one implementation, the perspective of the rendering is automatically rotated to align with respect to the plane of the display device and the tracked pose of the viewpoint of the user. In one implementation, the rotation of the rendering is computed according to the described projection matrix. In one implementation, the perspective of the rendering is computed according to: mirrorTworld · Mflip · worldTmirror · p.

[0027] In one implementation, the field of view of the virtual camera is automatically updated in response to movement of the tracked pose of the user’s point of view. In one implementation, the display device includes a fixed feature, and wherein the field of view of the virtual camera includes a boundary feature. In one implementation, the control system is configured to fit the boundary feature to coincide with the fixed feature. In one implementation, the control system fits the fixed feature for any given virtual position of the virtual camera that is automatically updated in response to movement of the tracked pose of the user’s point of view.

[0028] In one implementation, the common coordinate system includes an X-axis, a Y-axis, and a Z-axis, and wherein the virtual position of the virtual camera and the pose of the user’s point of view are equidistant from the plane of the display device with respect to each of the X-axis, the Y-axis, and the Z-axis.

[0029] In one implementation, the virtual position of the virtual camera is automatically updated to move toward the plane of the display device in response to movement of the tracked pose of the user’s point of view toward the plane of the display device. In one implementation, the virtual position of the virtual camera is automatically updated to move away from the plane of the display device in response to movement of the tracked pose of the user’s point of view away from the plane of the display device.

[0030] In one implementation, the control system renders the computer image to increase in size in response to movement of the tracked pose of the user’s point of view toward the plane of the display device. In one implementation, the control system renders the computer image to decrease in size in response to movement of the tracked pose of the user’s point of view away from the plane of the display device.

[0031] In one implementation, the computer image of the physical object is derived from a 3D model. In one implementation, the control system is configured to control the display device to display one or more slices of the 3D model. In one implementation, the control system does so as a function of the tracked pose of the physical object, the display device, and / or the user’s point of view.

[0032] In one implementation, the one or more slices are rendered from a perspective based on the field of view of the virtual camera having the virtual position that is automatically updated in response to movement of the tracked pose of the user’s point of view.

[0033] In one implementation, the control system is configured to control the display device to automatically change the one or more slices to other slices. In one implementation, the control system does so in response to one or more of: a tracked pose of the user’s viewpoint; a tracked pose of the display device; a tracked pose of the physical object; and / or a tracked pose of a surgical instrument.

[0034] In one implementation, the navigation system includes a head-mounted device worn by the user. In one implementation, the navigation system is configured to track the pose of the user’s viewpoint by being configured to track a pose of the head-mounted device. In one implementation, the navigation system includes a camera. In one implementation, the camera is configured to face the user. In one implementation, the navigation system is configured to track the pose of the user’s viewpoint by being configured to track a pose of the user’s head, face, or eyes using the camera. In one implementation, the camera is mounted to the display device. In one implementation, the camera is mounted to another device. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 An example of a surgical system that can be used with the spatially aware display techniques described herein is shown.

[0036] Figure 2 is a mockup illustration of one implementation and setup of fixed view frustum visualization techniques, where the perspective of the displayed image depends on the pose of the display device and the pose of a physical object (e.g., a patient) relative to the display device.

[0037] Figure 3A is an illustrative real-world example of one implementation of fixed view frustum visualization techniques taken from a first-person viewpoint, showing a user manually adjusting a display device to adjust the visualization of a 3D model of a physical object.

[0038] Figure 3B and Figure 3C provides a mockup illustration of one implementation of fixed view frustum visualization techniques, comparatively showing adjustments to the pose of a display device and virtual camera position.

[0039] Figure 4A and Figure 4B is an illustration showing an implementation of screen parallel slice visualization techniques, where slices of a 3D model of an object are obtained on a display device that is parallel to the plane of the display device and displayed.

[0040] Figure 5A and Figure 5Bis an illustrative real-world example of a screen parallel slice visualization technique in use with a fixed frustum visualization technique from a first-person perspective, which comparatively shows a user changing the pose of the display device, which causes the slice to correspondingly change perspective.

[0041] Figure 6 is an illustrative real-world example of an implementation of a screen parallel slice visualization technique, in which the slice is displayed in relation to a 3D model of a physical object (e.g., a patient) according to the tracked pose of the tool.

[0042] Figure 7 is an illustrative real-world example of an implementation of a screen parallel slice visualization technique, in which the displayed image is modified to illustrate the spatial layering of the object, tool, and slice relative to each other, so as to simulate their actual spatial positions in the real coordinate system in which these items exist.

[0043] Figure 8 is an implementation of an adjustment system configured to adjust the pose of a display device, for example for use with a fixed frustum visualization technique.

[0044] Figure 9 is a simulated diagram of an implementation and setup of a dynamic mirrored frustum visualization technique, in which the perspective of the displayed image depends on the pose of the display device, the pose of the object, and the pose of the user’s viewpoint, so that the displayed image can be rotated and sized to match the user’s viewpoint.

[0045] Figure 10A and Figure 10C provides a simulated diagram of an implementation of a dynamic mirrored frustum visualization technique, which comparatively shows changes in the pose of the respective user viewpoint and virtual camera position from a third-person perspective.

[0046] Figure 10B and Figure 10D respectively show the environment of Figure 10A and Figure 10C from a first-person perspective.

[0047] Figure 10E and Figure 10F is an illustrative real-world example of an implementation of a dynamic mirrored frustum visualization technique from a first-person perspective, and comparatively shows changes in the user’s perspective relative to the display device, which causes the displayed image to rotate and size to match the user’s viewpoint.

[0048] Figures 11A-11CThis diagram illustrates an implementation of dynamic mirrored frustum visualization technology in various views within the X, Y, Z coordinate system. It compares three scenarios, showing changes to the pose of the corresponding user viewpoint and the position of the virtual camera, as well as rotating the image rendering to align with the user's perspective.

[0049] Figure 12A and Figure 12B This is an illustrative real-world example of an implementation of viewpoint-oriented slice visualization using dynamic mirrored frustum visualization techniques, taken from a first-person perspective. It comparatively demonstrates how changing the user's viewpoint relative to the display device causes the displayed slice to change its perspective to align with the user's viewpoint.

[0050] Specific implementation method

[0051] I. Overview of Surgical Systems

[0052] refer to Figure 1 The image shows a surgical system 10, which can be used with a spatially aware display as described in Section 2 below. System 10 is used to treat a target site or anatomical volume A of a patient 12, such as bone or soft tissue. Figure 1 In the middle, patient 12 is undergoing surgical procedures. Figure 1 The anatomical structures included in the procedure are the femur (F), pelvis (PEL), and tibia (T) of patient 12. Surgical procedures may involve tissue removal or other forms of treatment. Treatment may include cutting, coagulation, damaging tissue, other in-situ tissue treatments, etc. In some examples, the surgical procedure involves partial or total knee or hip replacement surgery, shoulder replacement surgery, spinal surgery, or ankle surgery. In some examples, system 10 is designed to remove material to be replaced by a surgical implant, such as hip and knee implants, including single-chamber knee implants, double-chamber knee implants, multi-chamber knee implants, or total knee implants, acetabular cup implants, femoral stem implants, screws, anchors, other fasteners, etc. Some of these types of implants are illustrated in U.S. Patent Application Publication No. 2012 / 0330429 entitled “Prosthetic Implant and Method of Implantation,” the disclosure of which is hereby incorporated by reference. The system 10 and techniques disclosed herein can be used to perform other surgical or non-surgical procedures, or for industrial or other applications.

[0053] The system 10 can include a robotic manipulator 14, also referred to as a surgical robot. The manipulator 14 has a base 16 and a plurality of links 18. A manipulator cart 17 supports the manipulator 14 such that the manipulator 14 is fixed to the manipulator cart 17. The links 18 collectively form one or more arms (e.g., robotic arms) of the manipulator 14. The manipulator 14 can have a serial arm configuration (as shown in Figure 1

[0054] In the example shown, the manipulator 14 includes a plurality of joints J and a plurality of joint encoders 19 located at the joints J for determining position data of the joints J. For simplicity, only one joint encoder 19 is shown in Figure 1 Figure 1 The manipulator 14 according to one example has six joints J1-J6 that realize at least six degrees of freedom (DOF) of the manipulator 14. However, the manipulator 14 can have any number of degrees of freedom and can have any suitable number of joints J and can have redundant joints.

[0055] The manipulator 14 does not necessarily require joint encoders 19, but can instead or in addition use motor encoders present on motors at one or more or each joint J. Additionally, the manipulator 14 does not necessarily require revolute joints, but can instead or in addition use one or more prismatic joints. Any suitable combination of joint types is contemplated.

[0056] The base 16 of the manipulator 14 is generally a part of the manipulator 14 that provides a fixed frame of reference for the manipulator 14 or other components of the system 10 generally. In general, the origin of the manipulator frame of reference MNPL is defined at the fixed reference of the base 16. The base 16 can be defined relative to any suitable part of the manipulator 14, such as one or more of the links 18. Alternatively or in addition, the base 16 can be defined relative to the manipulator cart 17, such as at the location where the manipulator 14 is physically attached to the cart 17. In one example, the base 16 is defined at the intersection of the axes of joints J1 and J2. Thus, although joints J1 and J2 are moving parts in reality, the intersection of the axes of joints J1 and J2 is still a virtual fixed reference pose that provides a fixed position and orientation reference and does not move relative to the manipulator 14 and / or the manipulator cart 17.

[0057] ​​In some examples, manipulator 14 can be a handheld manipulator in which base 16 is a base portion of the tool (e.g., the portion held by hand by the user) and a tool tip is movable relative to the base portion. The base portion has a tracked reference coordinate frame and the tool tip has a tool tip coordinate frame that is computed (e.g., via motors and / or joint encoders and forward kinematics) relative to the reference coordinate frame. Movement of the tool tip can be controlled to follow a path because its pose relative to the path can be determined. Such a manipulator 14 is shown in U.S. Patent No. 9,707,043, entitled “Surgical Instrument Including Housing, A Cutting Accessory that Extends from the Housing and Actuators that Establish the Position of the Cutting Accessory Relative to the Housing,” filed August 31, 2012, which is hereby incorporated by reference herein.

[0058] Manipulator 14 and / or manipulator cart 17 house a manipulator controller 26 or other type of control unit. Manipulator controller 26 can include one or more computers, or any other suitable form of controller that directs the motion of manipulator 14. Manipulator controller 26 can have a central processing unit (CPU) and / or other processors, memory (not shown), and storage (not shown). Manipulator controller 26 is loaded with software as described below. The processor(s) can include one or more processors for controlling the operation of manipulator 14. The processor(s) can be any type of microprocessors, multi-processors, and / or multi-core processing systems. Manipulator controller 26 can additionally or alternatively include one or more microcontrollers, field programmable gate arrays, system on a chip, discrete circuits, and / or other suitable hardware, software, or firmware capable of carrying out the functions described herein. The term processor(s) is not intended to limit any implementation to a single processor. Manipulator 14 can also include a user interface UI having one or more displays and / or input devices (e.g., buttons, sensors, switches, keyboards, mice, microphones (voice activation), gesture control devices, touchscreens, joysticks, foot pedals, etc.).

[0059] The surgical tool 20 is coupled to the manipulator 14 and is movable relative to the base 16 to interact with the anatomy in certain modes. In certain embodiments, the tool 20 is or forms part of an end effector 22 supported by the manipulator 14. The tool 20 can be grasped by a user. One possible arrangement of the manipulator 14 and tool 20 is described in U.S. Patent No. 9,119,655, filed August 2, 2013, entitled "Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes," the disclosure of which is hereby incorporated by reference. The manipulator 14 and tool 20 can be arranged in alternative configurations. The tool 20 can be similar to the tool shown in U.S. Patent Application Publication No. 2014 / 0276949, filed March 15, 2014, entitled "End Effector of a Surgical Robotic Manipulator," which is hereby incorporated by reference. In addition to or as an alternative to the manipulator 14 and tool 20, a separate handheld surgical tool can be used.

[0060] The tool 20 includes an energy applicator 24 designed to contact tissue of the patient 12 at a target site. In one example, the energy applicator 24 is a burr 25. The burr 25 can be spherical and include a spherical center, a radius (r), and a diameter. Alternatively, the energy applicator 24 can be a drill bit, a saw blade 27 (see Figure 1

[0061] ​The tool 20 can include a tool controller 21 to control operation of the tool 20, such as to control power of the tool 20 (e.g., to control power of a tool drive, such as a rotary motor of the tool 20), to control movement of the tool 20, to control irrigation / aspiration of the tool 20, and / or the like. The tool controller 21 can be in communication with the manipulator controller 26 or other components. The tool 20 can also include a user interface UI having one or more displays and / or input devices (e.g., buttons, triggers, sensors, switches, keyboards, mice, microphones (voice activation), gesture control devices, touchscreens, joysticks, foot pedals, etc.) coupled to the tool controller 21, the manipulator controller 26, and / or other controllers described herein. The manipulator controller 26 controls a state (e.g., position and / or orientation) of the tool 20 (e.g., of a TCP) relative to a coordinate system, such as the manipulator coordinate system MNPL. The manipulator controller 26 can control a velocity (linear or angular), acceleration, or other motion derivative of the tool 20.

[0062] In one example, a tool center point (TCP) is a predetermined reference point defined at the energy applicator 24. The TCP has a known or calculable (i.e., not necessarily static) pose relative to other coordinate systems. The geometry of the energy applicator 24 is known in or defined relative to the TCP coordinate system. The TCP can be located at the spherical center of the bur 25 or at the distal end of the saw blade 27, such that only one point is tracked. The TCP can be defined in various ways depending on the configuration of the energy applicator 24. The manipulator 14 can employ joint / motor encoders or any other non-encoder position sensing method to enable determination of the pose of the TCP. The manipulator 14 can use joint measurements to determine the TCP pose and / or can employ techniques to directly measure the TCP pose. Control of the tool 20 is not limited to a center point. For example, the tool 20 can be represented using any suitable primitive, mesh, or the like.

[0063] The system 10 also includes a navigation system 32. One example of the navigation system 32 is described in U.S. Patent No. 9,008,757, filed September 24, 2013, entitled “Navigation System Including Optical and Non-Optical Sensors,” which is hereby incorporated by reference. The navigation system 32 tracks movement of various objects. Such objects include, for example, the manipulator 14, the tool 20, and anatomical structures (e.g., the femur F, the pelvis PEL, and the tibia T). The navigation system 32 tracks these objects to gather state information of the objects relative to a (navigation) localizer coordinate system LCLZ. Coordinates in the localizer coordinate system LCLZ are convertible to the manipulator coordinate system MNPL, other coordinate systems, and / or vice versa using transformations.

[0064] The navigation system 32 may include a trolley assembly 34 housing a navigation controller 36, and / or other types of control units. The navigation user interface (UI) communicates operationally with the navigation controller 36. The navigation user interface includes one or more displays 38. The navigation system 32 is capable of displaying a graphical representation of the relative status of the tracked object to the user using the one or more displays 38. The navigation user interface (UI) also includes one or more input devices to input information into the navigation controller 36 or otherwise select / control certain aspects of the navigation controller 36. Such input devices include interactive touchscreen displays. However, input devices may include any one or more of buttons, keyboards, mice, microphones (voice-activated), gesture control devices, foot switches, etc.

[0065] The navigation system 32 also includes a navigation locator 44 coupled to the navigation controller 36. In one example, the locator 44 is an optical locator and includes a camera unit 46 having an external housing 48 that houses one or more optical sensors 50. The locator 44 may include its own locator controller 49 and may also include a camera VC.

[0066] Navigation system 32 includes one or more trackers. In one example, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker 54, a second patient tracker 55, and a third patient tracker 56. Figure 1 In the example shown, the manipulator tracker is securely attached to tool 20 (i.e., tracker 52A), the first patient tracker 54 is securely attached to the femur F of patient 12, the second patient tracker 55 is securely attached to the pelvis PEL of patient 12, and the third patient tracker 56 is securely attached to the tibia T of patient 12. In this example, patient trackers 54, 55, and 56 are securely attached to the skeletal portion. The pointer tracker PT is securely attached to the pointer P, which is used to align the anatomical structures with the locator coordinate system LCLZ. Manipulator trackers 52A and 52B may be attached to any suitable component of manipulator 14 other than tool 20, such as base 16 (i.e., tracker 52B), or any one or more links 18 of manipulator 14. Trackers 52A, 52B, 54, 55, 56, and PT may be secured to their respective components in any suitable manner. For example, the tracker can be rigidly fixed, flexibly connected (optical fiber), or physically spaced (e.g., ultrasound), as long as there is a suitable (supplementary) way to determine the relationship (measurement result) between the corresponding tracker and the object associated with it.

[0067] Any one or more of the trackers can include an active marker 58. The active marker 58 can include a light emitting diode (LED). Alternatively, the trackers 52A, 52B, 54, 55, 56, PT can have a passive marker, such as a reflector that reflects light emitted from the camera unit 46. Other suitable markers not specifically described herein can be used.

[0068] The localizer 44 tracks the trackers 52A, 52B, 54, 55, 56, PT to determine the states of the trackers 52A, 52B, 54, 55, 56, PT, which correspond to the states of the objects to which they are attached, respectively. The localizer 44 can perform known triangulation techniques to determine the states of the trackers 52, 54, 55, 56, PT and the associated objects. The localizer 44 provides the states of the trackers 52A, 52B, 54, 55, 56, PT to the navigation controller 36. In one example, the navigation controller 36 determines the states of the trackers 52A, 52B, 54, 55, 56, PT and communicates them to the manipulator controller 26. As used herein, the state of an object includes, but is not limited to, data defining the position and / or orientation of the tracked object or an equivalent / derivative of the position and / or orientation. For example, the state can be the pose of the object and can include linear velocity data, and / or angular velocity data, etc.

[0069] The navigation controller 36 can include one or more computers, or any other suitable form of controller. The navigation controller 36 has a central processing unit (CPU) and / or other processor, memory (not shown), and storage (not shown). The processor can be any type of processor, microprocessor, or multi-processor system. The navigation controller 36 is loaded with software. For example, the software translates signals received from the localizer 44 into data representing the position and orientation of the tracked objects. The navigation controller 36 can additionally or alternatively include one or more microcontrollers, field programmable gate arrays, system on a chip, discrete circuits, and / or other suitable hardware, software, or firmware capable of carrying out the functions described herein. The term processor is not intended to limit any implementation to a single processor.

[0070] While one example of the navigation system 32 is shown employing triangulation techniques to determine the state of the objects, the navigation system 32 can have any other suitable configuration for tracking the manipulator 14, the tool 20, and / or the patient 12. In another example, the navigation system 32 and / or the localizer 44 is ultrasound-based. For example, the navigation system 32 can include an ultrasound imaging device coupled to the navigation controller 36. The ultrasound imaging device images any of the aforementioned objects (e.g., the manipulator 14, the tool 20, and / or the patient 12) and generates state signals to the navigation controller 36 based on the ultrasound images. The ultrasound images can be 2-D, 3-D, or a combination of both. The navigation controller 36 can process the images in near real-time to determine the state of the objects. The ultrasound imaging device can have any suitable configuration and can be different from Figure 1 the camera unit 46 shown.

[0071] In another example, the navigation system 32 and / or the localizer 44 is radio frequency (RF)-based. For example, the navigation system 32 can include an RF transceiver coupled to the navigation controller 36. The manipulator 14, the tool 20, and / or the patient 12 can include RF transmitters or transponders attached thereto. The RF transmitters or transponders can be passive or actively powered. The RF transceiver transmits RF tracking signals based on RF signals received from the RF transmitters and generates state signals to the navigation controller 36. The navigation controller 36 can analyze the received RF signals to associate relevant states therewith. The RF signals can have any suitable frequency. The RF transceiver can be positioned in any suitable location to effectively track the objects using the RF signals. Moreover, the RF transmitters or transponders can have any suitable structural configuration, which can be substantially different from Figure 1 the trackers 52A, 52B, 54, 55, 56, PT shown.

[0072] In yet another example, the navigation system 32 and / or the localizer 44 is electromagnetic-based. For example, the navigation system 32 can include an EM transceiver coupled to the navigation controller 36. The manipulator 14, the tool 20, and / or the patient 12 can include EM components attached thereto, such as any suitable magnetic trackers, electromagnetic trackers, inductive trackers, and the like. The trackers can be passive or actively powered. The EM transceiver generates an EM field and generates state signals to the navigation controller 36 based on EM signals received from the trackers. The navigation controller 36 can analyze the received EM signals to associate relevant states therewith. Again, this example of the navigation system 32 can have a structural configuration different from Figure 1 the navigation system 32 configuration shown.

[0073] The navigation system 32 can have any other suitable components or structures not specifically listed herein. Moreover, any of the techniques, methods, and / or components described above with respect to the illustrated navigation system 32 can implement or provide any of the other examples of the navigation system 32 described herein. For example, the navigation system 32 can use only inertial tracking or any combination of tracking techniques, and can additionally or alternatively include optical fiber-based tracking, machine vision tracking, and the like. While in some implementations we use optical IR tracking, the concepts and techniques described herein can be used to work with any sufficiently accurate 6D tracking technology. Moreover, we assume that existing surgical navigation systems already contain appropriate methods for preoperatively registering preoperative image data and surgical plans with the patient.

[0074] The system 10 includes a control system 60 that includes any one or more of the manipulator controller 26, the navigation controller 36, and the tool controller 21, among other components. The control system 60 includes one or more software programs and software modules. The software modules can be part of one or more programs that operate on the manipulator controller 26, the navigation controller 36, the tool controller 21, or any combination thereof to process data to assist in the control of the system 10. The software programs and / or modules include computer-readable instructions stored in a non-transitory memory 64 on the manipulator controller 26, the navigation controller 36, the tool controller 21, or a combination thereof to be executed by one or more processors 70 of the controller 21, 26, 36. The memory 64 can be any suitable memory configuration, such as RAM, non-volatile memory, etc., and can be implemented locally or from a remote database. Additionally, software modules for prompting and / or communicating with a user can form part of one or more programs and can include instructions stored in the memory 64 on the manipulator controller 26, the navigation controller 36, the tool controller 21, or any combination thereof. A user can interact with any of the input devices of the navigation user interface UI or other user interface UI to communicate with the software modules. The user interface software can run on a device separate from the manipulator controller 26, the navigation controller 36, and / or the tool controller 21.

[0075] The control system 60 can include any suitable configuration of inputs, outputs, and processing devices suitable to implement the functions and methods described herein. The control system 60 can include the manipulator controller 26, the navigation controller 36, or the tool controller 21, or any combination thereof, or can include only one of these controllers. These controllers can communicate via a wired bus or communication network, via wireless communication, or otherwise. The control system 60 can also be referred to as a controller. The control system 60 can include one or more microcontrollers, field programmable gate arrays, system on a chip, discrete circuits, sensors, displays, user interfaces, indicators, and / or other suitable hardware, software, or firmware capable of implementing the functions described herein.

[0076] II. Spatially-aware display technology

[0077] Systems, methods, and techniques related to spatially-aware displays for computer-assisted interventions are described herein. A new display and visual interaction paradigm is presented that aims to reduce the complexity of understanding the spatial transformation between the user's (e.g., surgeon's) viewpoint, physical objects (e.g., patient), 2D and 3D data (e.g., preoperative, intraoperative patient data), and tools during computer-assisted interventions. An intervention display, such as in a surgical navigation system, can be registered with both the patient and the surgeon's perspective. With this technique, the surgeon can maintain his / her own direct perspective on the patient, independent of any need for additional displays or direct perspective augmentation. In some implementations, the monitor used in the operating room is registered with the patient and the surgeon's viewpoint. This enables the physician to effortlessly relate their perspective on the tools and patient to the virtual representation of the patient data. The surgeon's direct perspective on the patient and his / her workspace can remain unchanged. The position and orientation of the display are an integral part of the visualization pipeline. Therefore, the pose of the display is tracked dynamically with respect to other objects of interest, such as the patient, the instruments, and in some implementations, the surgeon's head. This information is then used as input to the image-guided surgery visualization user interface.

[0078] At least two implementations are presented. The first implementation uses a "Fixed View Frustum" that relates the pose of the display to the patient and the tools. For the Fixed View Frustum technique, the display is tracked, for example, by adding tracking markers and calibrating the spatial relationship between the physical display and the tracking markers.

[0079] The second implementation is built on the mirror metaphor and extends the first technique to also incorporate the pose of the surgeon's head as another parameter into the visualization pipeline, in which case the display will be associated with a "dynamic mirror frustum of view". The visualization technique of tracking the surgeon's viewpoint for a dynamic mirror frustum. Estimation of the surgeon's viewpoint can be achieved by using head tracking targets or by mounting a camera to the surgical display and combining existing video-based head pose estimation algorithms. Once the tracking information is available in a common global coordinate system, we can compute the spatial relationships between the patient, the surgical display, the tools and the surgeon's viewpoint by deriving the relevant transformations from the tracked entities' spatial relationships.

[0080] These new display and visual exploration paradigms aim at reducing the complexity of understanding the spatial transformations between the user's viewpoint, the physical object (O), the preoperative / intraoperative 2D and 3D data and the surgical tools 110, 20 during computer-aided interventions with minimal current setup changes. Any surgical tracking system can be used to track the display, the tools and the user's head that are integrated into the computer-aided intervention system. The proposed solutions allow the physician to effortlessly relate their view of the tools and the patient to the virtual data on the surgical monitor. Independently of the need for an interactive device such as a mouse or a joystick, the user can obtain the possibility to interact with the patient data by intuitively moving their viewing position and observing the patient data from different perspectives related to the patient's position.

[0081] A. Fixed Frustum of View Visualization Technique

[0082] Reference Figures 2-8 An exemplary visualization method that can be used with the surgical system 10 includes a fixed frustum of view (FVF) for helping a user to interact with a physical object (O). The physical object (O) can be an anatomical structure as shown in the drawings, or any other object that requires interaction, setup or intervention, such as a surgical device, a robotic device, a surgical training model of an anatomical structure, and the like. For simplicity, the anatomical structure is shown as an object, however, the concept is not limited thereto.

[0083] One or more external screens or display devices 100 are provided. These display devices 100 can be those displays 38 on the navigation cart 34 assembly or any other external display that is spaced apart from the user and not worn by the user. The display devices 100 can be any suitable type of display, including but not limited to: LED, LCD, OLED, touch screen, holographic; and can display any type of image. The display devices 100 can also take any suitable geometric shape, including rectangular, square, circular, and the like.

[0084] During use, the one or more display devices 100 are located on a side (SI) of the physical object (O) opposite a side (S2) on which a user is located. In other words, the physical object (O) is between the user's point of view (V) and the display device 100. Thus, the display device 100 used herein is distinguished from a head-mounted display or a flat screen between the user's point of view (V) and the object (O). Here, the physical object (O) is shown as a virtual representation for illustrative purposes only.

[0085] Of course, the user is not prohibited from moving between the physical object (O) and the display device 100. However, the implementation of the spatially-aware display takes into account the practical reality that a surgeon desires to visualize the physical object (O) on the side on which he / she is currently located, and typically the side of the physical object (O) opposite the surgeon will provide an inverted perspective. This will be further understood below in the context of a field of view of a virtual camera facing forward toward the display device 100.

[0086] The display device 100 defines a plane (P). The plane (P) is defined as parallel to or coincident with the actual front face of the display device 100. Here, the plane (P) is a virtual object for computational purposes, as will be described below. The control system 60, including the navigation system 32, which includes any one or more of the controllers described herein, is configured to register computer images (R) with the physical object (O). The registration of the physical object (O) is not limited to any technique and can be performed according to any suitable method, including digitization of the physical object (O), non-contact registration (e.g., ultrasound), 2D / 3D image registration using imaging devices (e.g., CT, X-ray), and the like. The computer images (R) can represent virtual models (VM) of any portion of the physical object (O). These computer images (R) can be rendered on the display device 100. These computer images (R) can be static or dynamic and can include actual / real video graphics images, mixed reality, augmented reality, virtual reality images or models, or any combination thereof.

[0087] The control system 60 tracks the pose of the physical object (O) using any of the patient tracking techniques described above through the positioner 44, including but not limited to the patient trackers 54, 55, 56. In some implementations, the positioner 44 can also track the pose of the display device 100 in a common coordinate system with the physical object (O). The display device 100 can be tracked using a display tracker 102 or by any other suitable technique such as those described above. For example, the display device 100 can be tracked using machine vision (e.g., with or without a tracker), using optical or non-optical techniques. Alternatively or additionally, when equipped with a motorized adjustment system, the pose of the display device 100 can be determined based on kinematic data and independent of the positioner 44, as will be described below.

[0088] The control system 60 controls the display device 100 to render the registered computer image (R) at least in dependence on the tracked pose of the physical object (O) and the display device 100. Thus, the rendering of the computer image (R) depends at least on the pose of the physical object (O) and the pose of the display device 100. In particular, with reference to Figure 2 , the perspective of the rendering is based on the viewpoint or field of view (FOV) of a virtual camera (VC) facing the plane (P) of the display device 100. The field of view (FOV) can be based on a pinhole camera model projected towards the display device 100. The field of view (FOV) is also referred to as the view frustum (F), which will be described below.

[0089] The virtual position (VP) of the virtual camera (VC) is located on a side (SI) of the physical object (O) opposite to a side (S2) of the display device 100. Since the virtual camera (VC) is located Figure 2 in front of the plane (P) of the display device 100, the computer image (R) shows a side of the physical object (O) on the side (SI) and not a mirrored view on the side (S2).

[0090] The virtual position (VP) can be located at a predetermined distance (d) from the plane (P) of the display device 100. The distance (d) is based on a Z-axis drawn between the virtual position (VP) of the virtual camera (VC) and the plane (P). The Z-axis can be defined with reference to the virtual position (VP) or the display device 100. The virtual position (VP) can be located at any position on the virtual camera (VC) and can be defined by a single point or multiple points or any type of surface or volume geometry. In some examples, the virtual position (VP) remains fixed at the predetermined distance (d). Alternatively, the predetermined distance (d) can change in response to certain conditions. For example, a user can specify a preference for the distance (d) to the control system 60. The distance (d) can change automatically depending on conditions such as, but not limited to: the type of surgical procedure, a certain step of the procedure, the tracked pose of the viewpoint (V) of the user, the tracked pose of the object, the tracked pose of the display device 100, the tracked pose of the tool 110, 20, etc. In one example, the distance (d) can be defined at any suitable distance and within any suitable range, including but not limited to between 1 meter and 5 meters, or any distance included therebetween.

[0091] In one example, the virtual position (VP) of the virtual camera (VC) is transverse to the plane (P) of the display device 100. In other words, a Z-axis drawn between the virtual position (VP) and the plane (P) of the display device 100 defines an angle with respect to the plane (P), where the angle is in the range between 0 and 180 degrees. In a more particular implementation, the virtual position (VP) of the virtual camera (VC) is orthogonal to the plane (P) of the display device 100. In other words, a line drawn between the virtual position (VP) and the plane (P) of the display device 100 is 90 degrees perpendicular to the plane (P). In other words, in this example, the computer image (R) is rendered with an orthographic perspective projection from a given distance (d) that is orthogonal to the display device 100.

[0092] The virtual position (VP) of the virtual camera (VC) also has an X-coordinate position and a Y-coordinate position with respect to the plane (P) of the display device 100. The X- coordinate and Y-coordinate can be defined with reference to the virtual position (VP) or the display device 100. In one example, the X-coordinate and Y-coordinate are defined in a plane that is parallel to the plane (P) of the display device 100. However, the X-coordinate and Y-coordinate plane need not be parallel to the plane (P), for example, if the virtual camera (VC) is projected toward the display device 100 at a non-orthogonal angle.

[0093] Furthermore, whether the projection is transverse or orthogonal to the display device 100, the line drawn between the virtual position (VP) and the plane (P) can be toward the geometric center of the display device 100. Alternatively, the line drawn between the virtual position (VP) and the plane (P) can be offset from the geometric center of the display device 100. For example, the offset position can be toward a corner or edge of the display device 100. The X-Y arrangement of the virtual position (VP) with respect to the display device 100 can change in response to certain conditions. For example, a user can specify a preference for the X-Y arrangement of the virtual position (VP) to the control system 60. The X-Y arrangement of the virtual position (VP) can be automatically changed according to conditions, such as but not limited to: the type of surgical procedure, a certain step of the procedure, a tracked pose of the user’s viewpoint (V), a tracked pose of an object, a tracked pose of the display device 100, a tracked pose of the surgical tool 20, and so on. In one example, the distance (d) can be defined at any suitable distance and within any suitable range, including but not limited to between 1 meter and 5 meters, or any distance included therebetween.

[0094] Rendering of a computer image (R) based on a virtual camera (VC) is achieved using a view frustum (F) originating from the virtual camera (VC) and passing through an object, which in this case is the plane (P) of the display device 100. The view frustum (F) can be a spatial region in a 3D modeled world that can be rendered on the display device 100 and can be considered as the field of view of the virtual camera (VC). The apex of the frustum (F) is the zero point originating from the virtual camera (VC) and can also be considered as the virtual position (VP) of the virtual camera (VC). In one example, the plane (P) of the display device 100 can be located at the base (B) or far clipping plane (FCP) of the view frustum (F). The view frustum (F) can include a near clipping plane (NCP) proximate to the virtual camera (VC). The far clipping plane (FCP) and the near clipping plane (NCP) cut the frustum (F) almost perpendicular to the viewing direction, such that objects closer to the camera than the near clipping plane (NCP) or beyond the far clipping plane (FCP) are not rendered on the computer image (R). For processing efficiency reasons, objects that are partially or completely outside the view frustum (F) can be removed from the rendering process. The computer image (R) on the display device 100 is projected based on the view frustum (F). The view frustum (F) can be based on any planar truncation or any suitable 3-dimensional shape, including a pyramid, a cone, and the like. The view frustum (F) can take any configuration other than that shown in the figures or described herein.

[0095] In FVF technology, the virtual position (VP) of the virtual camera (VC) is automatically updated by the control system 60 in response to (manual or motorized) adjustments to the pose of the display device 100. If the user moves or rotates the display device 100, the position of the virtual camera (VC) is automatically updated. Thus, in contrast to standard visual display within a surgical navigation system, the psychological mapping of real objects to their image on the screen is simplified.

[0096] An example of such a fixed view frustum display visualization is shown in Figures 3A-3C An illustrative real-world example is shown in Figure 3A and two simulations are shown in Figure 3B and Figure 3C Figure 3A An operator is shown moving the display device 100 with their hand to visualize the internal virtual model (VM) of the physical object (O). As the user adjusts the display device 100, the computer rendering (R) on the display device 100 is updated accordingly. Figure 3B and Figure 3C provide simulated illustrations of FVF technology from two different viewpoints of the same configuration of the virtual camera (VC). As the comparison shows, the display device 100 in Figure 3B and Figure 3C ​between the pose is changed and the virtual position (VP) of the virtual camera (VC) is updated accordingly. The computer image (R) is also changed in perspective according to the relative positioning between the view frustum (F) and the physical object (O).

[0097] i. Screen Parallel Slice Visualization (SPSV)

[0098] Reference is now made to Figures 4 to Figure 6 A screen parallel slice visualization (SPSV) sub-technique of the FVF method is described. While FVF visualization is intuitive to use and can display 3D data, the technique can be further implemented using slice visualization. Thus, a slice view can be incorporated into the described spatially-aware visualization concept.

[0099] Reference is now made to Figure 4A This can be implemented in one implementation by slicing a 3D virtual model (VM) of the physical object (O) into a plurality of slices (SL1, SL2, SL3... SLN). In one implementation, the slices (SL) are made at planes parallel to the plane (P) of the display device 100. The virtual model (VM) can be a CT model, an X-ray model, an MRI model, or a model created using any other type of imaging technology. Alternatively, instead of slicing a 3D virtual model (VM), the imaging data of the physical object (O) can include a plurality of slices that can be obtained from computer memory, whether or not they have been combined into a 3D model.

[0100] In one implementation, and with reference to Figure 4B The displayed slice (SL) can be based on a specified plane (p) that slices through the view frustum (F). The specified plane (p) can be fixed relative to the virtual camera (VC) position, or the specified plane (p) can be dynamically changed based on any of the conditions described herein. When the field of view of the virtual camera (VC) passes through the physical object (O), the control system 60 can immediately obtain the slice (SL) that corresponds to the intersection of the specified plane (p) and the physical object (O).

[0101] The orientation of the computer image (R) of the slice (SL) can be adjusted based on the pose of the display device 100. As shown in Figure 5A compared to Figure 5B The user is given an interactive method for rotating the display device 100, which in turn rotates the slice (SL) and the virtual model (T) of the tool 110, 20. The physical object (O) and the tool 110, 20 are in the same relative position in Figure 5A and Figure 5B

[0102] Alternatively or additionally, as Figure 6 ​As shown, the orientation of the slice (SL) can be set based on the tool 110, 20 being within the viewing frustum (F) of the virtual camera (VC). The pose of the tool 110, 20 can be tracked by the localizer 44 using any suitable means, such as those described herein. The rendered computer image (R) can include an image of the tool 110, 20 and the slice (SL) using the fixed frustum method as discussed above. Alternatively or additionally, the orientation of the slice (SL) can be set based on the perspective of the user as defined by the tracked pose of the viewpoint tracking system (VTS), such as the user-facing external camera (C) or the head-mounted device 120 (described in detail below).

[0103] During use of these techniques, the displayed slice (SL) can dynamically change to a different slice (SL). The slice (SL) can change as a function of the pose of the physical object (O), the display device 100, or any combination thereof. Additionally or alternatively, the slice (SL) can be changed using any of the input devices described herein. Furthermore, as Figure 6 shown, the tool 110, 20 can be used to change the slice (SL). For example, the pose, position, and / or distance or orientation of the tool 110, 20 relative to the physical object (O) or the display device 100 can automatically cause the control system 60 to change the slice (SL). In some implementations, portions of the slice (SL) can be displayed based on a (2D or 3D) virtual boundary (VB) associated with the tip 114 of the tool 110, 20, as Figure 4B shown. For example, the virtual boundary (VB) can be defined by any shape (e.g., rectangular, box-shaped, circular, or spherical) having any suitable dimensions (e.g., 200 mm diameter), with the tool tip 114 located at the center of the boundary (VB). If the tool 110, 20 is moved toward the object (O) such that the object (O) intersects the virtual boundary (VB), the control system 60 can dynamically present or change the displayed slice (SL) corresponding to the intersection. The slice (SL) can be presented in its entirety (as Figure 4B shown) or can be cropped according to the location of the virtual boundary (VB) ( Figure 6 ).

[0104] ii. Background-accurate rendering

[0105] Referring to Figure 7 , the control system 60 is configured to render the computer image (R) in a background-accurate manner. In other words, the computer image (R) rendering takes into account the spatial layering of the object (O), the tool 110, 20, and / or the slice (SL) relative to one another in a common coordinate system so as to simulate their actual spatial positions in the real coordinate system in which these items exist. As Figure 7As shown, tools 110 and 20 are physically inserted into the object (O), and the display device 100 presents a virtual model (T) of tools 110 and 20 and corresponding slices (SL) of the 3D model (VM) of the object (O). However, as shown, these renderings (R) are layered based on the perspective of a virtual camera (VC). In other words, the virtual model (T) of tools 110 and 20 is intentionally obscured by corresponding portions of the 3D model (VM) of the object (O). In this case, the thoracic cavity of the anatomical structure obscures the axis of tools 110 and 20. Similarly, slices (SL) are displayed such that portions of the slices (SL) are obscured by corresponding portions of the 3D model (VM) of the object (O). In this case, slices (SL) are obscured by several rib portions. Although this technique obscures the visualized portion, it is beneficial for the user to visualize the environment in a context closely resembling the real-world spatial positioning of the object (O), slices (SL), and tools 110 and 20. This visualization technique can be used with any of the techniques described herein (including FVF and SPSV techniques), as well as with any of the specific implementations, conditions, and / or situations described herein.

[0106] iii. Adjust the system

[0107] like Figure 8 As shown, the display device 100 may optionally be connected to the adjustment system 104. The adjustment system 104 may be passive or active and may include any features or possible configurations of the robot manipulator 14 described above. For example, the adjustment system 104 may include multiple links (L), joints (J), and actuators (M) for driving the joints (J). The adjustment system 104 may be manually or automatically controlled to adjust the pose of the display device 100. The pose of the display device 100 may be moved in up to six degrees of freedom (three translational and three rotational). Sensors (S) may be mounted on any of the components of the adjustment system 104 to detect the movement of the adjustment system 104. The control system 60 may use the measurements from the sensors (S) to kinematically derive the pose of the display device 100. This may also be done in addition to using the positioner 44, or as an alternative. The sensor(s) can be any suitable configuration, including but not limited to: motor current sensors, joint position sensors or encoders (rotational, absolute, incremental, virtual absolute, etc.), optical sensors, inertial sensors (accelerometers, inclinometers, gyroscopes, etc.), etc. Any of the sensors(s) may also be mounted on the display device 100 itself.

[0108] In some cases, such as Figure 8As shown, one or more user input devices 106 can be wired or wirelessly connected to the control system 60 to enable a user to control the pose of the display device 100. The input devices 106 include, but are not limited to: a foot pedal 106a, a handheld teach pendant 106b, a display input 106c (such as a display of a tablet, smartphone, or navigation system), a tool 110, 20, and / or a viewpoint tracking system (VTS) (such as a user-facing external camera (C) or a head-mounted device 120). The input devices can also be any of those described above, including but not limited to: buttons, sensors, switches, keyboards, mice, microphones (voice activation), gesture control devices, touchscreens, joysticks, etc. The input devices 106 receive commands from a user, and the control system 60 directs one or more actuators M to adjust the joints J and ultimately the pose of the display device 100 according to the commands.

[0109] The tracked tool 110, 20 can also be used to trigger the control system 60 to control the adjustment system 104. In one implementation, the pose of the tracked tool 110, 20 (whether it be position and / or orientation or any derivative thereof (velocity, acceleration, etc.)) can cause the control system 60 to adjust the pose of the display device 100. For example, the tracked pose of the tool 110, 20 can be compared to the virtual position (VP) of a virtual camera (VC), the frustum (F) of view, the plane (P) of the display device 100, or any combination thereof. The control system 60 can evaluate this comparison relative to a threshold condition or measurement. If the control system 60 determines that the tool 110, 20 has moved in a manner that satisfies or exceeds the threshold condition, the control system 60 can command the adjustment system 104 to adjust the display device 100. This can be beneficial for various situations, including but not limited to: the tool 110, 20 moving toward or away from an object (O), the tool 110, 20 moving in / out of the frustum (F) of view, holding the tool 110, 20 within the field of view of a virtual camera (VC), etc.

[0110] With continued reference to Figure 8The head-mounted device 120 can also be used to trigger the control system 60 to control the adjustment system 104. In one implementation, the pose of the head-mounted device 120 is tracked using any of the tracking techniques described herein or equivalents thereof. The tracked pose of the head-mounted device 120, whether it be a position and / or orientation or any derivative thereof (velocity, acceleration, etc.), can cause the control system 60 to adjust the pose of the display device 100. For example, the tracked pose of the head-mounted device 120 can be compared to a virtual position (VP) of a virtual camera (VC), a frustum of view (F), a plane (P) of the display device 100, or any combination thereof. The control system 60 can evaluate this comparison relative to a threshold condition or measurement. If the control system 60 determines that the head-mounted device 120 has moved in a manner that satisfies or exceeds the threshold condition, the control system 60 can instruct the adjustment system 104 to adjust the display device 100. This can be beneficial for various situations, including but not limited to: the head-mounted device 120 moving towards or away from the object (O), the head-mounted device 120 moving in / out of the frustum of view (F), keeping the head-mounted device 120 within the angle of view of the virtual camera (VC), etc.

[0111] Additionally or alternatively, any other view tracking system (VTS), such as a user-facing external camera (C), can be used to track the user's view (V) for triggering the control system 60 to control the adjustment system 104.

[0112] The adjustment system 104 can be used in any of the techniques described herein, including the FVF and SPSV techniques, as well as for any of the particular implementations, conditions, and / or situations described herein.

[0113] B. Dynamic Mirrored Frustum Visualization Technique

[0114] Reference is made to Figures 9-1 2. Another exemplary visualization method that can be used with the surgical system 10 includes a dynamic mirrored frustum (DMVF) for helping a user to interact with a physical object (O).

[0115] As will be appreciated from the following description, there are technical similarities between the FVF technique and the DMVF technique. Accordingly, any and all of the above description related to the system 10, the FVF technique, and any of the physical, computational, and / or technical aspects associated therewith are fully incorporated by reference for use with and can be applied by the DMVF technique described herein, and thus are not repeated for the sake of simplicity of description.

[0116] For the DMVF technique, and with reference to Figure 9The display device 100 defines a plane (P). The physical object (O) is located in front of the plane (P). In other words, the physical object (O) is located in front of the displayed screen of the display device 100.

[0117] A user viewing the display device 100 is also located in front of the plane (P). The viewpoint (V) of the user is tracked using a viewpoint tracking system (VTS). In one implementation, the viewpoint tracking system (VTS) includes a camera (C) facing the user. The viewpoint tracking system (VTS) can be the navigation system itself, or can be part of the navigation system or separate from the navigation system. The camera (C) can be the localizer 44 or a separate device. The camera (C) can be mounted to the display device 100 or other location. The control system 60 can receive signals from the camera (C) and use a pose estimation algorithm to identify changes in the position and / or orientation of the user's face, eyes, head, or other features.

[0118] In another implementation, the viewpoint tracking system (VTS) additionally or alternatively includes a head-mounted device 120 disposed directly on the user. The head-mounted device 120 is located in front of the plane (P) and in front of the displayed screen of the display device 100. In one configuration, the head-mounted device 120 is located on a first side (SI) of the physical object (O), and the display device 100 is located on a second, opposite side (S2) of the physical object.

[0119] The head-mounted device 120 includes one or more trackable features (HT) such that its pose is trackable by the navigation system 32 and / or the control system 60. The trackable features (HT) can be any of the types described above or any equivalent thereof. In this technology, the head-mounted device 120 is any device configured to be able to track the pose of the user's viewpoint. Here, pose means the position of the head-mounted device 120, and optionally the position and orientation. The user's viewpoint can be defined by the overall field of view of the user's vision, the direction in which the user turns his / her head, and / or the gaze direction of the user's eyes. The head-mounted device 120 can be or include one or more trackers attached to any part of the user's head and / or eyes or headwear such as eyeglasses (such as, for example, those shown), goggles, headbands, contact lenses, and the like. Figure 8 In one example, the head-mounted device 120 is an optical see-through head-mounted display. Other examples of the head-mounted device 120 are contemplated. In the example shown, for simplicity, the viewpoint originating from the tracked head-mounted device 120 is shown as a sphere and the viewpoint is facing the display device 100. Figure 9

[0120] ​As with the FVF technique, the navigation system 32 and / or the control system 60 register the computer images (R) with the physical object (O). The navigation system 32 and / or the control system 60 track the pose of the physical object (O), the display device 100, and the user’s viewpoint (V) relative to each other in a common coordinate system. The display device 100 is controlled for rendering the registered computer images (R) in accordance with the tracked poses of the physical object (O), the display device 100, and the user’s viewpoint (V). In other words, the pose of each of these items can influence how the computer images (R) are displayed.

[0121] The perspective of the computer images (R) is based on the field of view or frustum of view (F) of a virtual camera (VC) having a virtual position (VP) located behind the plane (P), which is shown in Figure 9 as a side (S3) located behind the display device 100. The virtual camera (VC) faces the back of the display device 100 and towards the user’s viewpoint originating from the viewpoint tracking system (VTS). The physical object (O), the display device 100, and the user’s viewpoint (V) are at least partially located within the frustum of view (F) in Figure 9 . Since the virtual camera (VC) is located behind the plane (P) of the display device 100, the rendering (R) provides a visualization as if the display device 100 is a mirror relative to the user’s viewpoint. In Figure 9 , the computer images (R) show a side of the physical object (O) at a side (S2).

[0122] The virtual position (VP) of the virtual camera (VC) is automatically updated in response to adjustments to the tracked pose of the user’s viewpoint (V). The user can move left or right or forward and backward and see the physical object (O) and any tools 110, 20 move correspondingly on the mirror-like rendering (R) of the display device 100. By knowing the poses of the physical object (O), the display device 100, and the user’s viewpoint (V), the control system 60 is able to create the computer images (R) so that they follow the same laws as a real mirror.

[0123] Figures 10A-10F Examples of the DMVF technique are shown. In particular, a simulated mirror view is shown in Figures 10A-10D and real-world illustrative examples are shown in Figure 10E and Figure 10F . In contrast to the third-person perspective, as shown in Figure 10A and Figure 10C , the virtual position (VP) of the virtual camera (VC) changes according to the user’s viewpoint (V) originating from the viewpoint tracking system (VTS) (shown as a sphere). Figure 10B Examples of the DMVF technique are shown. In particular, a simulated mirror view is shown in Figure 10Aa first-person point of view of the content that will be seen on the display device 100 in the scene. Figure 10C shows a first-person point of view of the content that will be seen on the display device 100 in the scene. Figure 10D shows a first-person point of view of the content that will be seen on the display device 100 in the scene.

[0124] The display device 100 shows the object in front of the display as a mirror would do taking into account the pose of the screen, the object and the user's point of view. This paradigm works can help a more intuitive visualization of the physical object (O) by using motion parallax and observing the structure from another desired point of view. The user is free to interact with the data just by looking without necessarily needing an interactive device like a mouse or a joystick. Since the user does not always need to redefine their perspective on the object (O), the user is able to turn the interactivity on or off, for example, with any input device such as a foot pedal. The user can change the visualization hands-free. Since the natural human visual system is adapted to observe real mirror images, the DMVF visualization provides an intuitive interface. DMVF also helps to explore and define the optimal slice (SL) or rendering (R) for a given navigation task. The slice (SL) or rendering (R) can remain fixed during the surgical action until further data exploration is needed.

[0125] i. Virtual camera and mirror frustum settings for DMVF

[0126] The virtual position (VP) of the virtual camera (VC) is dynamically changed in the DMVF method according to at least the tracked pose of the user's point of view (V). In one implementation, the point of view of the virtual camera (VC) can be derived as described herein. In one example, the control system 60 moves the virtual camera (VC) according to a projection matrix that is adjusted so that the viewing frustum (F) is fitted so that the boundary features (BF) of the viewing frustum (F) coincide with the features (DF) of the display device 100. The features (DF) of the display device 100 include points or other geometric shapes that encode the size, aspect ratio, position and orientation of the display device 100 in the common or world coordinate system. In Figure 9 In the example where the viewing frustum (F) comprises a truncated quadrangular pyramid and the display device 100 is rectangular, the boundary features (BF) of the viewing frustum (F) are points on the edges of the sides of the pyramid and the features (DF) are the fixed corners of the display device 100. Thus, for any position movement of the virtual camera (VC) in the coordinate system, the points on the edges of the viewing frustum (F) are made to coincide with the corners of the display device 100. Of course, this configuration can vary depending on the geometry of the viewing frustum (F) and the display device 100. This results in a warped image (R) that appears as a mirror reflection from the tracked pose of the user's point of view (V) with the correct perspective.

[0127] According to the virtual position (VP) of the virtual camera (VC), the mirror world position p of the viewpoint 镜像 The mirror can be computed as follows: the normal of the mirror plane (P) is defined as (0; 0; 1). In its local coordinate system, the mirror corresponds to the multiplication by the matrix M 翻转 multiplied by the matrix expressed in [1] and taking into account the fixed features (DF) of the display device 100 (i.e. in this case the four corners).

[0128] [1]

[0129] This is equivalent to a scaling by -1 in the z-direction. Let us assume, according to one implementation, that in order to provide a mirror view, the rendering depends on the position of the user's viewpoint (V) relative to the mirror plane (P) and not on its orientation. A first transformation is performed to transform the world position of the virtual camera (VC) viewpoint p into the mirror's local coordinates. This results in the frustum (F) being rotated to the user's viewpoint. Then the mirror is performed with the matrix M 翻转 providing a perspective projection relative to the display device 100. Finally, the mirror's mirror local coordinates are transformed back to the world position of the virtual camera (VC) viewpoint p, which translates the user's viewpoint (V) to the top of the viewing frustum (F). The result is the projection matrix expressed by:

[0130] [2]

[0131] In the equation expressed in [2], the notation is used to denote a transformation from coordinate system A to B. This computation enables the base (B) of the frustum (F) or far clipping plane (FCP) to be realigned or rotated from its other XY coordinate aligned position relative to the display device 100 to an aligned position located relative to the user's viewpoint (V). Thereafter, the user's viewpoint aligned base (B) or far clipping plane (FCP) realignment is back to the XY coordinates of the display device 100 so that an off-axis projection matrix can be applied to apply a perspective rendering of the computer image (R).

[0132] To enable the computer image (R) on display device 100 to render a mirror view of a physical object (O) or tool 110, 20 in front of display device 100 in relation to the user's pose, an off-axis projection matrix may be employed in one implementation. Off-axis here means that the line drawn from the user's viewpoint (V) to display device 100 will not be at the geometric center of display device 100, but rather off-center. In this case, the viewing frustum (F) becomes asymmetrical and its shape changes as the user's viewpoint (V) changes. The position of the user's viewpoint (V) relative to the plane (P) of display device 100 may be monitored by control system 60. Since the shape of the frustum (F) changes according to the user's viewpoint (V), the boundary features (BF) of the viewing frustum (F) also change. In one implementation, the viewing frustum (F) may be recalculated and / or updated frequently at intervals (e.g., every frame) because the pose of display device 100 and / or the user's viewpoint may change.

[0133] The off-axis projection matrix has a mirrored viewpoint p located at the vertex of the viewing frustum (F) and a base (B) matching the surface or plane (P) of the display device 100. The viewing frustum (F) is rotated to align with the user's viewpoint (V). This takes into account the position of the viewpoint and distorts the image so that it appears as a projection onto an oblique plane relative to the plane (P) of the display device 100. Figure 10C , Figure 10D In comparison, it is possible Figure 10A , Figure 10B The effect of the user's changing viewpoint on the frustum (F) and the resulting image distortion is observed. This implementation relies on a graphics API to set the projection matrix, where the frustum (F) is defined by near clipping plane (NCP) coordinates in view space, which is then rotated to be non-perpendicular and moved to a mirrored viewpoint. In other words, the base (B) of the frustum (F) dynamically modifies the XY plane of the plane (P) rotated out of the display device 100 and positioned to correspond to the angle of the user's viewpoint (V) derived from the viewpoint tracking system (VTS).

[0134] The calculations provided above offer an implementation of a virtual camera (VC) setup for a DMVF. However, alternative methods may exist that dynamically change the virtual position (VP) of the virtual camera (VC) at least according to the user's viewpoint (V) without departing from the scope of this concept. For example, the base (B) of a frustum (F) can be aligned with the display device 100. The boundary feature (BF) of the visible frustum (F) may exceed the geometry of the display device 100. In such cases, the control system 60 may limit the boundary feature (BF) of the visible frustum (F) relative to the geometry of the display device 100.

[0135] Figures 11A-11C The relative positioning between a virtual camera (VC) and a frustum (F), a user's viewpoint (V), a display device 100, and an object (O) in three example scenarios is compared in various views of an X, Y, Z coordinate system. In these examples, the position of the virtual camera (VC) and the user's viewpoint (V) can be mirrored individually with respect to the X, Y, and Z axes relative to the plane (P) of the display device 100. For example, the virtual camera (VC) and the user's viewpoint (V) are at substantially equal distances from the display device 100 in each of the X, Y, and Z directions. When visualized from the X-Y plane, the position of the virtual camera (VC) and the user's viewpoint (V) appear to coincide due to this mirroring effect. However, perfect mirroring is not required in all implementations, and the distances of (VC) and (V) to the display device 100 can not be equal for each of the axes and such distances can be customized or varied.

[0136] In Figure 11A , the user's viewpoint (V) is coaxial with the display device 100 at a given position. The base (B) of the frustum (F) of the virtual camera (VC) is aligned with the X-Y plane of the display device 100. The projected rendering is of a computer image (R) centered and aligned with the user's central viewpoint.

[0137] In Figure 11B , the user's viewpoint (V) is moved to the left of center (from the user's perspective) and is closer to the display device 100 than the given position of Figure 11A . This relative change in the motion of the user's viewpoint (V) causes the virtual camera (VC) to correspondingly change. In other words, the virtual camera (VC) is moved to the right and is moved further away from the display device 100 (from the virtual camera's perspective). This positions the virtual camera (VC) correctly with respect to the user's viewpoint (V). The viewing frustum (F) is rotated (counterclockwise from above) out of the X-Y plane of the display device 100 to align with the user's off-center viewpoint (V). In some cases, the modification to the frustum (F) causes the far clipping plane (FCP) to move further away from the apex of the frustum (F) in response to the position of the virtual camera (VC) moving further away from the display device 100. The projected rendering is of a computer image (R) that is rotated to align with the user's off-center viewpoint. The computer image (R) also renders objects such as a physical object (O) within the frustum (F) to appear further away from the display device 100 than Figure 11A compared in size because the position of the user's viewpoint (V) is further away from the display device 100.

[0138] In Figure 11C , the user's viewpoint (V) is moved to the right (from the user's perspective) and is further away from the display device 100 than the given position of Figure 11Athe given position of the user's viewpoint (V) moves closer to the display device 100. This relative change in the motion of the user's viewpoint (V) causes the virtual camera (VC) to move left and closer to the display device 100 (from the perspective of the virtual camera). This positions the virtual camera (VC) correctly with respect to the user's viewpoint (V). The frustum (F) is rotated out of the X-Y plane of the display device 100 (clockwise from above) to align with the user's viewpoint (V). In some cases, the modification to the frustum (F) causes the far clipping plane (FCP) to move closer to the apex of the frustum (F) in response to the position of the virtual camera (VC) moving away from the display device 100. The projection renders the computer image (R) rotated to align with the user's off-center viewpoint. The computer image (R) also renders objects such as a physical object (O) within the frustum (F) to appear to be at the same distance from the user's viewpoint (V) as they are from the display device 100. Figure 11A than the size of the user's viewpoint (V) is closer to the display device 100.

[0139] The dimensions of the components of FIG. 11 are provided for illustrative purposes only and can not be entirely to scale. Moreover, while the X-Z plane is shown to illustrate lateral motion between the user's viewpoint and the virtual camera (VC), the principles described herein can apply entirely to vertical motion between the two in the X-Y plane.

[0140] ii. Viewpoint-facing slice visualization

[0141] Referring now to Figure 12A and Figure 12B , a viewpoint-facing slice visualization (VFSV) sub-technique of the DMVF method is described. While DMVF visualization is intuitive to use and can display 3D data, it is envisioned to further incorporate slice views into the described spatially-aware DMVF visualization concept.

[0142] Slices of data can be taken according to any of the techniques described above with respect to the SPSV technique and illustrated with respect to Figure 4A and Figure 4B , and thus are not repeated herein for the sake of simplicity. Moreover, the selection of the displayed slice (SL) can be taken according to any of the techniques described above with respect to the SPSV technique and illustrated with respect to Figure 4A and Figure 4B .

[0143] In one implementation, the tool 110, 20 can be used to change the slice (SL). For example, the pose, position, and / or distance or orientation of the tool 110, 20 relative to the physical object (O) or the display device 100 can automatically cause the control system 60 to change the slice (SL). In some implementations, portions of the slice (SL) can be displayed based on a (2D or 3D) virtual boundary (VB) associated with the tip 114 of the tool 110, 20, as shown in Figure 4B For example, the virtual boundary (VB) can be defined by any shape (e.g., rectangular, box-shaped, circular, or spherical) having any suitable size (e.g., 200 mm diameter), with the tool tip 114 located at the center of the boundary (VB). If the tool 110, 20 is moved toward the object (O) such that the object (O) intersects the virtual boundary (VB), the control system 60 can dynamically present or change the displayed slice (SL) corresponding to the intersection. The slice (SL) can be presented in its entirety (as shown in Figure 4B ) or can be cropped according to the location of the virtual boundary (VB) ( Figure 6 ).

[0144] Additionally or alternatively, the slice (SL) can change according to the pose of the physical object (O), the display device 100, the user’s viewpoint, or any combination thereof. Any of the input devices 106 described herein can be used to change the slice (SL).

[0145] The orientation of the slice (SL) can be manipulated according to several different techniques. In a VFSV technique, the control system 60 tracks the display device 100 and the user’s viewpoint (V) according to the DMVF technique. The position of the slice (SL) is based on the tracked position of the tool 110, 20. However, the orientation of the slice (SL) is chosen to point toward the mirrored viewpoint. This guarantees that the slice (SL) faces the user when viewed through the mirror. That is, the slice (SL) is oriented to face the user’s viewpoint (V) as derived by the viewpoint tracking system (VTS). The orientation of the slice (SL) is rendered using the DMVF technique described above. That is, to accurately account for the user’s viewpoint (V) relative to the display device 100, the slice (SL) is rotated relative to the plane (P) of the display device 100 and projected to be smaller or larger.

[0146] Examples of the VFSV technique from two different first-person viewpoints can be seen in Figure 12A and Figure 12B , where the tool 110, 20 remains stationary between Figure 12A and Figure 12B . In Figure 12BIn this example, the user's viewpoint (V) shifts slightly to the left and moves closer to tools 110 and 20. The distance from the user's viewpoint (V) to the display device 100 remains approximately the same. In this example, the displayed slices (SL) appear identical to the user because the slices (SL) have been rotated to correspond to the lateral movement of the user's viewpoint (V). Figure 12B If a user is closer to the display device 100, the slice (SL) will appear larger accordingly, and vice versa.

[0147] Although the user’s viewpoint (V) is used to implement VFSV technology, the orientation of the computer image (R) of the slice (SL) can be adjusted based on the pose of the display device 100, the pose of the object (O), and / or the pose of the tracked tools 110, 20 within the view frustum (F) of the virtual camera (VC).

[0148] Additionally, the control system 60 is configured to render computer graphics (R) in a background-accurate manner for use with VFSV technology. In other words, the computer graphics (R) rendering takes into account the spatial layering of objects (O), tools 110, 20, and / or slices (SL) relative to each other in a common coordinate system, so as to simulate their actual spatial positions in the real coordinate system in which these items exist. Therefore, any of the above descriptions (including) related to background-accurate rendering for SPSV technology Figure 6 and Figure 7 The visualization can be applied in its entirety to VFSV technology and will not be repeated for the sake of simplicity.

[0149] C. Visualization Mode Switching

[0150] The control system 60 can switch between any of the visualization modes described herein at any time and in response to any input or condition. For example, the control system 60 can switch between any of the following: FVF and DMVF visualization; SPSV and VFSV visualization; 3D model visualization using FVF and SPSV technology (with or without a 3D model) (without slicing); 3D model visualization using DMVF technology (without slicing) and visualization using VFSV technology (with or without a 3D model); static visualization and any of FVF, SPSV, DMVF, and VFSV technologies.

[0151] Switching between visualization modes can be done according to user input. For example, control system 60 can receive commands from any of the input devices 106 described herein, including but not limited to: foot pedal 106a, handheld teach pendant 106b, display input 106c (such as a tablet, smartphone, display of a display device 100, or display of a navigation system), tool 110,20, and / or head mounted device 120, buttons, sensors, switches, keyboard, mouse, microphone (voice activation), gesture control device, touch screen, joystick, etc.

[0152] Switching between visualization modes can be done automatically according to conditions, such as but not limited to: type of surgical procedure, certain step of the procedure, tracked pose of the user’s viewpoint (V) or head mounted device 120, tracked pose of the object (O), tracked pose of the display device 100, tracked pose of the tool 110,20, or any combination thereof.

[0153] In the foregoing description, numerous implementing ways have been discussed. The implementing ways discussed herein, however, are not intended to be exhaustive or to limit the application to any particular form. The terminology used is intended to be in the nature of words of description rather than of limitation. Many modifications and variations are possible in light of the above teachings and it is

[0154] D. Case Study and Experimental Results

[0155] As a proof of concept, a confidential case study was conducted with three trauma surgeons (attending surgeon, resident surgeon, and resident physician). The setup included a 3D printed patient phantom, a display device 100 mounted on an adjustable table mount fixed to the OR table, and a tracking system 130 that tracked the patient phantom, tool 110,20, display device 100, and viewpoint. Flashpoint 6000 tracking system. 3D-printed patient phantoms contained segmented anatomy and corresponding CT volumes. During the experiment, surgeons explored the proposed techniques in a predefined order with a think-aloud protocol and no time limit, followed by a semi-structured interview. Surgeons were asked to pay attention to the differences in visualization methods and to consider possible application scenarios. First, surgeons could use conventional orthogonal slice visualization controlled by a tracked instrument. Next, surgeons were presented with the novel FVF and SPSV visualization techniques. Afterward, surgeons explored the head-tracked approach, the DMVF and VFSV visualization techniques. While using FVF and DMVF, surgeons were instructed to switch between two modes: segmented 3D structure and direct volume rendering. In the interview, surgeons praised the automated slice presentation and the idea of presenting them with images that matched their direct view to the patient's orientation. Overall feedback from the experts on the proposed concepts was very positive. Surgeons appreciated the fact that, for DMVF and VFSV, surgeons could interact with patient data in a sterile manner, which they considered important when interventions proved more complex than initially expected. Participants identified that the FVF and SPSV approach could be easily integrated into established surgical workflows. When asked about possible applications of the proposed concepts, answers included use cases in complex cranio-maxillofacial surgery, as well as interventions in orthopedic and trauma surgery, and interventions that make use of high precision, such as endoprosthetics. Finally, participants identified that the proposed visualization techniques could help to familiarize with a specific patient's anatomy and to understand how the instrument is currently positioned with respect to the planned trajectory. Participants unanimously identified that the novel visualization techniques helped with orientation regarding obtaining an overview of a specific patient's anatomy and how the instrument is currently positioned.

Claims

1. A system for facilitating interaction with physical objects, the system comprising: A display device, the display device defining a plane and located on a first side of the physical object; as well as A navigation system, coupled to a control system and configured to: Register the computer image with the physical object; Track the pose of the physical object and the display device in a common coordinate system; and Control the display device to render the registered computer image according to the physical object and the tracked pose of the display device; and The rendering perspective is based on a virtual camera, which has a virtual position located on a second side of the physical object opposite to the first side, and the virtual camera has a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in response to adjustments to the pose of the display device.

2. The system of claim 1, wherein the computer image of the physical object is derived from a 3D model, and the control system is configured to control the display device to display one or more slices of the 3D model according to the pose tracked by the physical object and the display device.

3. The system of claim 2, wherein the one or more slices are sliced ​​in a plane parallel to the plane of the display device.

4. The system of any one of claims 2-3, wherein the control system is configured to control the display device to automatically change the one or more slices to other slices in response to one or more of: the tracked pose of the display device; the tracked pose of the physical object; the tracked pose of a surgical instrument; and the tracked pose of a user's viewpoint.

5. The system of any one of claims 1-3, wherein the virtual position of the virtual camera is located at a predetermined distance from the plane of the display device, wherein the predetermined distance is automatically updated.

6. The system of claim 5, wherein the predetermined distance is automatically updated in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical instrument; the tracked pose of the user's viewpoint; the type of surgical procedure; and a specific step of the surgical procedure.

7. The system of any one of claims 1-3, wherein the virtual position of the virtual camera is located at a predetermined distance from the plane of the display device, and wherein the predetermined distance is fixed.

8. The system of any one of claims 1-3, wherein the virtual position of the virtual camera is automatically updated relative to the XY arrangement of the plane of the display device.

9. The system of claim 8, wherein the XY arrangement is automatically updated in response to at least one or more of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical instrument; the tracked pose of the user's viewpoint; the type of surgical procedure; and a specific step of the surgical procedure.

10. The system of any one of claims 1-3, wherein the virtual position of the virtual camera is fixed in the XY arrangement relative to the plane of the display device.

11. The system of any one of claims 1-3, wherein the pose of the display device is manually adjustable, and the virtual position of the virtual camera is automatically updated in response to manual adjustment of the pose of the display device.

12. The system as claimed in any one of claims 1-3, comprising: One or more actuators are coupled to the display device, and the control system is configured to control the one or more actuators to adjust the pose of the display device.

13. The system of claim 12, comprising: An input device is coupled to the control system, and the control system is configured to receive commands from the input device and control the one or more actuators to adjust the pose of the display device according to the commands.

14. The system of claim 12, comprising: A surgical instrument comprising one or more trackable features, wherein the navigation system is configured to track the pose of the surgical instrument in a common coordinate system and control the display device to display an image of the surgical instrument, and wherein the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument.

15. The system of claim 12, comprising: A viewpoint tracking system coupled to the navigation system and configured to track the pose of a user's viewpoint in a common coordinate system, wherein the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the user's viewpoint.

16. The system of claim 3, wherein the line defining the virtual position of the virtual camera and the plane of the display device is transverse to the plane of the display device.

17. The system of claim 3, wherein the line defining the virtual position of the virtual camera and the plane of the display device is orthogonal to the plane of the display device.

18. The system of any one of claims 16-17, wherein the line falls at the geometric center of the display device.

19. The system of any one of claims 16-17, wherein the line falls at a location offset from the geometric center of the display device.

20. A computer-readable storage medium storing program instructions thereon, which, when executed by a processor, cause the processor to perform steps of a method for operating a system to facilitate interaction with a physical object, the system comprising: A display device, the display device defining a plane and located on a first side of the physical object; And a navigation system, said navigation system being coupled to the control system, said method comprising: Register the computer image with the physical object; Track the pose of the physical object and the display device in a common coordinate system; and Control the display device to render the registered computer image according to the physical object and the tracked pose of the display device; and The rendering perspective is based on a virtual camera, which has a virtual position located on a second side of the physical object opposite to the first side, and the virtual camera has a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in response to adjusting the pose of the display device.

21. A computer program product for assisting in interaction with physical objects, the computer program product being usable with a system, the system comprising: A display device, the display device defining a plane and located on a first side of the physical object; and a navigation system coupled to a control system, the computer program product including instructions configured, when executed by one or more processors, to: Register the computer image with the physical object; Track the pose of the physical object and the display device in a common coordinate system; and Control the display device to render the registered computer image according to the physical object and the tracked pose of the display device; and The rendering perspective is based on a virtual camera, which has a virtual position located on a second side of the physical object opposite to the first side, and the virtual camera has a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in response to adjustments to the pose of the display device.

Images