Spatial awareness display for computer-assisted intervention

Abstract

Described herein are systems and techniques for spatially aware displays for computer-assisted intervention. The fixed view frustum technique renders computer images on a display using a perspective based on a virtual camera with a field of view facing the display, and automatically updates the virtual camera's virtual position in response to adjustments to the display's pose. The dynamic mirror view frustum technique renders computer images on a display using a perspective based on the field of view of a virtual camera whose virtual position is behind the display device. The virtual camera's virtual position is dynamically updated in response to movements of a user's viewpoint, which is located in front of the display device. Slice visualization techniques for use with the fixed view frustum and dynamic mirror view frustum techniques are also described herein.

Description

Technical Field

[0001] Cross - reference to Related Applications This international patent application claims priority to U.S. Provisional Patent Application No. 63 / 036,559, filed on Jun. 9, 2020, the entire content of which is incorporated herein by reference.

Background Art

[0002] In the very early days of fluoroscopy, radiologists used a fluorescent hand - held screen placed within the beam emitted from an X - ray source passing through a patient. Since the image was observed immediately where it was created, there was a direct correlation among the X - ray source, the patient, the screen, and the observer. With the progress of imaging and display technologies, it has become possible to view static images or live videos of a patient's anatomical structures anywhere a monitor can be placed. This has made the placement of the patient and the display more practical and has brought the advantage of reducing the radiation dose to the interventionalist, but the intuitive recognition of the spatial arrangement among the X - ray source, the patient, the screen, and the observer has been lost.

[0003] CT scans or MRI scans, which are essentially 3D, are often visualized as a set of three orthogonal slices along the patient's anatomical axis or along the axis of an instrument. More recently, in many cases, 3D images have been rendered on the operating room display in addition to slice visualization. These are often volume renderings where the user can define and control the viewing point. Since many surgeons are well - trained in the interpretation of 2D images, they prefer 2D images over 3D graphic renderings, and thus this situation defines the current clinical standard.

[0004] Intraoperative navigation has been introduced since the early 1990s and utilizes various visualization techniques. Numerous systems display slices of pre-operative or intra-operative data with additional annotations. These annotations relate to the surgical plan that needs to be performed during the operation. Such annotations often attempt to visualize deep-seated anatomical targets and safe routes for the surgeon to reach them, based on pre-operative images. In such systems, the surgeon's position and the display's location are not considered when rendering the images. Therefore, the display's location does not affect the visualization or the observer's position.

[0005] A series of related works are often referred to as Augmented Reality windows (AR). Conventional techniques for medical in-situ augmentation involved a tracked translucent display incorporating a head tracker and stereo glasses. Such translucent displays included a half-silvered glass pane that reflected the image from a computer display. Other techniques addressed the same challenge of creating an AR window of a patient's anatomical structure by using a translucent display between the patient and the surgeon. However, in such conventional techniques, the translucent glass pane was replaced with an active-matrix LCD. Subsequent attempts revisited the problem of the translucent glass pane, which refused to project high-contrast images from two DLP projectors. Other systems featured a tracked mobile opaque screen, where the screen's position affected the visualization. Again, such conventional systems were positioned between the surgeon and the patient, displaying slice views of the anatomical structure. However, this system does not take the user's perspective into account; in other words, the image on the screen is merely a two-dimensional representation unrelated to the surgeon's viewpoint.

[0006] Elsewhere, AR visualizations inspired by dentists' techniques for examining patients' mouths without changing their viewpoint were presented. It was identified that, in some AR applications, it is impossible to rotate or move objects in such techniques. To provide a secondary viewpoint of virtual objects within the AR view, it was proposed to generate additional virtual mirroring views. A spatially tracked joystick was used to move virtual mirrors that reflect virtual data like a real mirror within the AR view of a head-mounted device (HMD). [Overview of the project] [Means for solving the problem]

[0007] This summary of the invention provides a simplified overview of the selected concepts that will be further described later in the forms for carrying out the invention. This summary of the invention is not intended to limit the scope of the claimed subject matter, nor does it necessarily identify each of the important or essential features of the claimed subject matter.

[0008] In the first embodiment, a system is provided for assisting interaction with a physical object, comprising a display device that defines a plane and is positioned on a first side of the physical object, and a navigation system connected to a control system, wherein the navigation system is configured to register a computer image to the physical object, track the poses of the physical object and the display device in a common coordinate system, and control the display device to render the registered computer image according to the tracked poses of the physical object and the display device, wherein the rendering perspective is based on a virtual camera whose virtual position is positioned on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in accordance with adjustments to the pose of the display device.

[0009] In a second embodiment, a navigation system for the system of the first embodiment is provided.

[0010] In a third aspect, a method is provided for operating a system to assist interaction with a physical object, the system comprising a display device that defines a plane and is positioned on a first side of the physical object, and a navigation system connected to a control system, the method comprising aligning a computer image to the physical object, tracking the poses of the physical object and the display device in a common coordinate system, and controlling the display device to render the aligned computer image according to the tracked poses of the physical object and the display device, the rendering perspective being based on a virtual camera whose virtual position is positioned on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and the virtual position of the virtual camera being automatically updated in accordance with adjustments to the pose of the display device.

[0011] In a fourth aspect, a computer program product is provided for assisting interaction with a physical object, the computer program product being available for use in a system comprising a display device that defines a plane and is positioned on a first side of a physical object, and a navigation system connected to a control system, the computer program product including instructions, which, when executed by one or more processors, are configured to perform: align a computer image to a physical object, track the poses of the physical object and the display device in a common coordinate system, and control the display device to render an aligned computer image according to the tracked poses of the physical object and the display device, the rendering perspective being based on a virtual camera whose virtual position is positioned on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and the virtual position of the virtual camera being automatically updated in response to adjustments in the pose of the display device.

[0012] In a fifth aspect, a system is provided for assisting interaction with physical objects, the system comprising a display device defining a plane, the physical object being positioned in front of the plane, and a navigation system connected to a control system, the navigation system being configured to align a computer image to the physical object, track the poses of the physical object, the display device, and the user's viewpoint in a common coordinate system, and control the display device to render the aligned computer image according to the tracked poses of the physical object, the display device, and the user's viewpoint, the rendering perspective being based on the field of view of a virtual camera whose virtual position is behind the plane, and the virtual position of the virtual camera being automatically updated in accordance with the movement of the tracked pose of the user's viewpoint.

[0013] In the sixth aspect, a navigation system for the system of the fifth aspect is provided.

[0014] A seventh aspect provides a method for operating a system to assist interaction with a physical object, the system comprising a display device defining a plane, the physical object being positioned in front of the plane, and a navigation system connected to a control system, the method including aligning a computer image to a physical object, tracking the poses of the physical object, the display device, and the user's viewpoint in a common coordinate system, and controlling the display device to render the aligned computer image according to the tracked poses of the physical object, the display device, and the user's viewpoint, the rendering perspective being based on the field of view of a virtual camera whose virtual position is behind the plane, and automatically updating the virtual position of the virtual camera in accordance with the movement of the tracked pose of the user's viewpoint.

[0015] In the eighth aspect, a computer program product is provided for assisting interaction with physical objects, the computer program product being available for use in a system, the system comprising a display device defining a plane, the physical object being positioned in front of the plane, and a navigation system connected to a control system, the computer program product including instructions, which, when executed by one or more processors, are configured to align a computer image to a physical object, track the poses of the physical object, the display device, and the user's viewpoint in a common coordinate system, and control the display device to render the aligned computer image according to the tracked poses of the physical object, the display device, and the user's viewpoint, the rendering perspective being based on the field of view of a virtual camera whose virtual position is behind the plane, and the virtual position of the virtual camera being automatically updated in accordance with the movement of the tracked pose of the user's viewpoint.

[0016] In the ninth aspect, a system, method, or computer program product for assisting interaction with a physical object is provided, comprising a display device defining a plane and a navigation system connected to a control system, wherein the navigation system is configured to align a computer image to a physical object, track the poses of the physical object and the display device in a common coordinate system, and control the display device to render the aligned computer image according to the tracked poses of the physical object and the display device, the rendering perspective being based on a virtual camera having a field of view facing the plane of the display device, the virtual position of the virtual camera being automatically updated in response to adjustments in the pose of the display device.

[0017] In a tenth aspect, a system, method, or computer program product for assisting interaction with a physical object is provided, comprising a display device defining a plane and a navigation system connected to a control system, wherein the navigation system is configured to align a computer image to a physical object, track the poses of the physical object, the display device, and the user's viewpoint in a common coordinate system, and control the display device to render the aligned computer image according to the tracked poses of the physical object, the display device, and the user's viewpoint, the rendering perspective being based on the field of view of a virtual camera, the virtual position of the virtual camera being automatically updated in accordance with the movement of the tracked pose of the user's viewpoint.

[0018] Any of the above embodiments can be combined in part or in whole.

[0019] Any of the above embodiments can be combined with any of the embodiments described below, either alone or in combination.

[0020] In one embodiment, a computer image of a physical object is derived from a 3D model. In one embodiment, a control system is configured to control a display device to display one or more slices of the 3D model. In one embodiment, the slices are displayed according to the tracked pose of the physical object and / or the display device (i.e., the physical object or the display device or both). In one embodiment, one or more slices are sliced ​​on a plane aligned with the plane of the display device. In one embodiment, the plane is parallel to the display device. In one embodiment, the control system is configured to control the display device to automatically change one or more slices to other slices according to at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical instrument, and / or the tracked pose of the user's viewpoint (i.e., the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical instrument, or the tracked pose of the user's viewpoint, or all of them).

[0021] In one embodiment, the virtual position of the virtual camera is positioned at a predetermined distance from the plane of the display device. In one embodiment, the predetermined distance is automatically updated. In one embodiment, the predetermined distance is automatically updated according to at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, and / or a specific step of the surgical procedure (i.e., the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, or a specific step of the surgical procedure, or all of them). In one embodiment, the predetermined distance is manually updated in response to user input. In one embodiment, the predetermined distance is fixed.

[0022] In one embodiment, the XY arrangement of the virtual position of the virtual camera relative to the plane of the display device is automatically updated. In one embodiment, the XY arrangement is automatically updated. In one embodiment, the XY arrangement is automatically updated according to at least one of the following: the tracked pose of the display device, the tracked pose of physical objects, the tracked pose of surgical tools, the tracked pose of the user's viewpoint, the type of surgical procedure, and specific steps of the surgical procedure. In one embodiment, the XY arrangement of the virtual position of the virtual camera relative to the plane of the display device is manually adjustable based on user input. In one embodiment, the XY arrangement of the virtual position of the virtual camera relative to the plane of the display device is fixed.

[0023] In one embodiment, the pose of the display device is manually adjustable. In one embodiment, the virtual position of the virtual camera is automatically updated in response to manual adjustments to the pose of the display device. In one embodiment, one or more actuators are connected to the display device. In one embodiment, a control system is configured to control one or more actuators to adjust the pose of the display device. In one embodiment, an input device is connected to the control system, and the control system is configured to receive commands from the input device and, according to the commands, control one or more actuators to adjust the pose of the display device. In one embodiment, a surgical instrument includes one or more trackable features, and a navigation system is configured to track the pose of the surgical instrument in a common coordinate system and to control the display device to display an image of the surgical instrument. In one embodiment, the control system is configured to control one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument. In one embodiment, a viewpoint tracking system is connected to the navigation system, and the viewpoint tracking system is configured to track the pose of the user's viewpoint in a common coordinate system. In one embodiment, the control system is configured to control one or more actuators to adjust the pose of the display device based on the tracked pose of the user's viewpoint.

[0024] In one embodiment, a line is defined between the virtual position of the virtual camera and the plane of the display device. In one embodiment, the line traverses the plane of the display device. In one embodiment, the line is perpendicular to the plane of the display device. In one embodiment, the line strikes the geometric center of the display device. In one embodiment, the line strikes a position offset from the geometric center of the display device.

[0025] In one embodiment, the rendering perspective is automatically rotated so as to align with the tracked pose of the user's viewpoint relative to the plane of the display device. In one embodiment, the rotation of the rendering is calculated according to the projection matrix being described. In one embodiment, the rendering perspective is calculated as follows.

Number

[0026] In one embodiment, the field of view of the virtual camera is automatically updated in response to the movement of the tracked pose of the user's viewpoint. In one embodiment, the display device includes fixed features and the field of view of the virtual camera includes boundary features. In one embodiment, the control system is configured to match the boundary features to the fixed features. In one embodiment, the control system matches the boundary features to the fixed features at any given virtual position of the virtual camera that is automatically updated in response to the movement of the tracked pose of the user's viewpoint.

[0027] In one embodiment, the common coordinate system includes an X-axis, a Y-axis, and a Z-axis, and the virtual position of the virtual camera and the pose of the user's viewpoint are equidistant from the plane of the display device with respect to each of the X-axis, Y-axis, and Z-axis.

[0028] In one embodiment, in response to the movement of the tracked pose of the user's viewpoint approaching the plane of the display device, the virtual position of the virtual camera is automatically updated to approach the plane of the display device. In one embodiment, in response to the movement of the tracked pose of the user's viewpoint moving away from the plane of the display device, the virtual position of the virtual camera is automatically updated to move away from the plane of the display device.

[0029] In one embodiment, the control system renders a computer image enlarged in size as the tracked pose movement of the user's viewpoint approaches the plane of the display device. In another embodiment, the control system renders a computer image shrinking in size as the tracked pose movement of the user's viewpoint moves away from the plane of the display device.

[0030] In one embodiment, a computer image of a physical object is derived from a 3D model. In one embodiment, a control system is configured to control a display device to display one or more slices of the 3D model. In one embodiment, the control system is configured in accordance with the tracked pose of the physical object, the display device, and / or the user's viewpoint (i.e., the physical object, the display device, or the user's viewpoint, or all of them).

[0031] In one embodiment, the perspective of one or more slices is rendered based on the field of view of a virtual camera, whose virtual position is automatically updated in response to the tracking pose movement of the user's viewpoint.

[0032] In one embodiment, the control system is configured to control the display device to automatically change one or more slices to other slices. In one embodiment, the control system is configured in such a way depending on one or more of the following: the tracked pose of the user's viewpoint, the tracked pose of the display device, the tracked pose of physical objects, and / or the tracked pose of surgical instruments (i.e., the tracked pose of the user's viewpoint, the tracked pose of the display device, the tracked pose of physical objects, or the tracked pose of surgical instruments, or all of them).

[0033] In one embodiment, the navigation system includes a head-mounted device worn by the user. In one embodiment, the navigation system is configured to track the pose of the user's gaze by being configured to track the pose of the head-mounted device. In one embodiment, the navigation system includes a camera. In one embodiment, the camera is configured to be oriented toward the user. In one embodiment, the navigation system is configured to track the pose of the user's gaze by being configured to track the pose of the user's head, face, or eyes using the camera. In one embodiment, the camera is mounted on a display device. In one embodiment, the camera is mounted on another device. [Brief explanation of the drawing]

[0034] [Figure 1] This specification shows examples of surgical systems that can be used in conjunction with the spatial recognition display techniques described herein. [Figure 2] This is a simulation example of one embodiment and setup of the Fixed View Frustum visualization technique, in which the perspective of the displayed image depends on the pose of the display device and the pose of the physical object (e.g., patient) relative to the display device. [Figure 3A] This is an example of a fixed-view frustum visualization technique that shows the real world from a first-person perspective, demonstrating that the visualization of 3D models of physical objects can be adjusted by the user manually adjusting the display device. [Figure 3B] This paper provides a simulation example of one embodiment of the fixed-view frustum visualization technique, comparing and demonstrating adjustments to the display device's pose and the virtual camera's position. [Figure 3C] This paper provides a simulation example of one embodiment of the fixed-view frustum visualization technique, comparing and demonstrating adjustments to the display device's pose and the virtual camera's position. [Figure 4A] This figure illustrates an embodiment of the Screen Parallel Slice Visualization technique, in which slices of a 3D model of an object are obtained and displayed on the display device parallel to the plane of the display device. [Figure 4B] This figure illustrates an embodiment of the Screen Parallel Slice Visualization technique, in which slices of a 3D model of an object are obtained and displayed on the display device parallel to the plane of the display device. [Figure 5A] This is an example of a screen parallel slice visualization technique used in conjunction with the fixed-view frustum visualization technique, showing the real world from a first-person perspective. It compares and demonstrates how the perspective obtained by the slice changes accordingly when the user changes the pose of the display device. [Figure 5B] This is an example of a screen parallel slice visualization technique used in conjunction with the fixed-view frustum visualization technique, showing the real world from a first-person perspective. It compares and demonstrates how the perspective obtained by the slice changes accordingly when the user changes the pose of the display device. [Figure 6] This is a real-world example of a screen parallel slice visualization technique, where slices of a 3D model of a physical object (e.g., a patient) are displayed depending on the pose of the tracked tool. [Figure 7] This is a real-world example of a screen parallel slice visualization technique, in which the displayed image is modified to account for the relative spatial layering of objects, tools, and slices, thereby simulating their actual spatial positions in the real-world coordinate system in which these items exist. [Figure 8] For example, this is one embodiment of an adjustment system configured to adjust the pose of a display device, for use in conjunction with the fixed-view frustum visualization technique. [Figure 9]This is a simulation example of one embodiment and setup of a dynamic mirror view frustum visualization technique, in which the perspective of the displayed image depends on the pose of the display device, the pose of the object, and the pose of the user's viewpoint, thereby allowing the displayed image to be rotated and resized to match the user's viewpoint. [Figure 10A] This provides a simulation example of one embodiment of the dynamic mirror view frustum visualization technique, comparing the changes in each user's viewpoint pose and the virtual camera's position from a third-person view. [Figure 10B] Figure 10A shows the environment as viewed from a first-person perspective. [Figure 10C] This provides a simulation example of one embodiment of the dynamic mirror view frustum visualization technique, comparing the changes in each user's viewpoint pose and the virtual camera's position from a third-person view. [Figure 10D] Figure 10C shows the environment as viewed from a first-person perspective. [Figure 10E] This is an example of a first-person perspective demonstration of the Dynamic Mirror View Frustum visualization technique, illustrating how the displayed image rotates and resizes to match the user's viewpoint as the user's perspective on the display device changes. [Figure 10F] This is an example of a dynamic mirror frustum visualization technique that shows the real world from a first-person perspective, illustrating by comparison how the displayed image rotates and is resized to match the user's viewpoint as the user's viewpoint on the display device changes. [Figure 11A] This figure illustrates one embodiment of the dynamic mirror view frustum visualization technique in various views of the X, Y, and Z coordinate systems, comparing the changes in the user's viewpoint pose and virtual camera position, as well as the rotation of the image rendering to align with the user's viewpoint, for three different scenarios. [Figure 11B]This figure illustrates one embodiment of the dynamic mirror view frustum visualization technique in various views of the X, Y, and Z coordinate systems, comparing the changes in the user's viewpoint pose and virtual camera position, as well as the rotation of the image rendering to align with the user's viewpoint, for three different scenarios. [Figure 11C] This figure illustrates one embodiment of the dynamic mirror view frustum visualization technique in various views of the X, Y, and Z coordinate systems, comparing the changes in the user's viewpoint pose and virtual camera position, as well as the rotation of the image rendering to align with the user's viewpoint, for three different scenarios. [Figure 12A] This is an example of a viewpoint-facing slice visualization technique used in conjunction with dynamic mirror-view frustum visualization, showing the real world from a first-person perspective. It compares and demonstrates how the displayed slices change perspective to align with the user's viewpoint as the user's viewpoint relative to the display device changes. [Figure 12B] This is an example of a viewpoint-facing slice visualization technique used in conjunction with dynamic mirror-view frustum visualization, showing the real world from a first-person perspective. It compares and demonstrates how the displayed slices change perspective to align with the user's viewpoint as the user's viewpoint relative to the display device changes. [Modes for carrying out the invention]

[0035] [I. Overview of the Surgical System] Referring to Figure 1, a surgical system 10 is shown that can be used with a spatially-aware display, as described in Section II below. System 10 is useful for treating a target site or anatomical volume A of patient 12, such as the treatment of bone or soft tissue. In Figure 1, patient 12 is undergoing a surgical procedure. The anatomical structures in Figure 1 include the femur F, pelvis PEL, and tibia T of patient 12. The surgical procedure may include tissue removal or other forms of treatment. Treatment may include tissue cutting, coagulation, injury, or other in-situ tissue treatment. In some embodiments, the surgical procedure may include partial or complete knee or hip replacement, shoulder replacement, spinal surgery, or ankle surgery. In some embodiments, System 10 is designed to excise material that will be replaced by surgical implants, such as hip and knee implants, including unicondylar knee implants, bicondylar knee implants, polycondylar knee implants, or total knee implants, acetabular cup implants, femoral shaft implants, screws, anchors, and other fasteners. Some of these types of implants are shown in U.S. Patent Application Publication No. 2012 / 0330429, entitled “Prosthetic Implant and Method of Implantation,” which is incorporated herein by reference. System 10 and techniques disclosed herein may be used to perform other surgical or nonsurgical procedures, or may be used in industrial or other applications.

[0036] System 10 may include a robotic manipulator 14, also known as a surgical robot. The manipulator 14 has a base 16 and a number of links 18. A manipulator cart 17 supports the manipulator 14 so that it is fixed to the manipulator cart 17. Collectively, the links 18 form one or more arms (e.g., robot arms) of the manipulator 14. The manipulator 14 may have a series arm configuration (shown in Figure 1), a parallel arm configuration, or any other suitable manipulator configuration. In other embodiments, multiple manipulators 14 may be used in multiple arm configurations.

[0037] In the embodiment shown in Figure 1, the manipulator 14 comprises a plurality of joints J and a plurality of joint encoders 19 positioned on the joints J to identify the position data of the joints J. For simplicity, only one joint encoder 19 is shown in Figure 1, but other joint encoders 19 may be shown in the same manner. According to one embodiment, the manipulator 14 has six joints J1 to J6 that implement at least six degrees of freedom (DOF) with respect to the manipulator 14. However, the manipulator 14 may have any number of degrees of freedom, any appropriate number of joints J, and may have redundant joints.

[0038] The manipulator 14 does not necessarily require a joint encoder 19, but may alternatively or additionally utilize one or more joints J or motor encoders present on each joint J's motor. Furthermore, the manipulator 14 does not necessarily require a rotary joint, but may alternatively or additionally utilize one or more prismatic joints. Any suitable combination of joint types is intended.

[0039] Typically, the base 16 of the manipulator 14 is part of the manipulator 14 that generally provides a fixed reference coordinate system to the manipulator 14 or other components of the system 10. Typically, the origin of the manipulator coordinate system MNPL is defined at the fixed reference of the base 16. The base 16 can be defined with respect to any suitable part of the manipulator 14, such as one or more of the links 18. Alternatively or additionally, the base 16 can be defined with respect to the manipulator carriage 17, such as when the manipulator 14 is physically attached to a carriage 17. In one embodiment, the base 16 is defined at the intersection of the axis of joint J1 and the axis of joint J2. Therefore, although joints J1 and J2 are actually movable components, the intersection of the axes of joints J1 and J2 is still a virtual fixed reference pose, which provides a fixed reference for both position and orientation and does not move relative to the manipulator 14 and / or the manipulator trolley 17 (i.e., the manipulator 14 or the manipulator trolley 17 or both).

[0040] In some embodiments, the manipulator 14 can be a handheld manipulator, where the base 16 is the base portion of the tool (e.g., the portion freely held by the user), and the tool tip is movable relative to the base portion. The base portion has a tracked reference coordinate system, and the tool tip has a tool tip coordinate system calculated with respect to the reference coordinate system (e.g., via a motor and / or joint encoder (i.e., a motor or a joint encoder or both), and calculations of forward kinematics). Since the pose of the tool tip can be specified with respect to a path, the movement of the tool tip can be controlled to follow a path. Such a manipulator 14 is shown in U.S. Patent No. 9,707,043, filed August 31, 2012, entitled "Surgical Instrument Including Housing, A Cutting Accessory that Extends from the Housing and Actuators that Establish the Position of the Cutting Accessory Relative to the Housing," the contents of which are incorporated herein by reference.

[0041] The manipulator 14 and / or the manipulator carriage 17 (i.e., the manipulator 14 or the manipulator carriage 17 or both) house a manipulator controller 26 or other type of control unit. The manipulator controller 26 may comprise one or more computers or any other suitable form of controller that directs the movement of the manipulator 14. The manipulator controller 26 may have a central processing unit (CPU) and / or other processors (i.e., a central processing unit (CPU) or other processors or both), memory (not shown), and storage (not shown). The manipulator controller 26 is loaded with software as described below. The processor may include one or more processors that control the operation of the manipulator 14. The processor may be any type of microprocessor, multiprocessor, and / or multicore processing system (i.e., any type of microprocessor, multiprocessor, or multicore processing system, or all of them). The manipulator controller 26 may additionally or alternatively comprise one or more microcontrollers, field-programmable gate arrays, system-on-a-chips, discrete circuits, and / or other suitable hardware capable of performing the functions described herein (i.e., one or more microcontrollers, field-programmable gate arrays, system-on-a-chips, discrete circuits, or other suitable hardware capable of performing the functions described herein, or all of them), software, or firmware. The term processor is not intended to limit any embodiment to a single processor. The manipulator 14 may also comprise a user interface (UI) with one or more displays and / or input devices (e.g., push buttons, sensors, switches, keyboards, mice, microphones (for voice control), gesture control devices, touchscreens, joysticks, foot pedals, etc.) (i.e., one or more displays or input devices, or both).

[0042] The surgical tool 20 is coupled to the manipulator 14 and is movable relative to the base 16 to interact with an anatomical structure in a specific mode. In certain embodiments, the tool 20 is an end effector 22 supported by the manipulator 14, or forms part thereof. The tool 20 can be grasped by a user. One possible arrangement of the manipulator 14 and the tool 20 is described in U.S. Patent No. 9,119,655, filed 2 August 2013, entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is incorporated herein by reference. The manipulator 14 and the tool 20 may be arranged in alternative configurations. Tool 20 may be similar to that shown in U.S. Patent Application Publication No. 2014 / 0276949, “End Effector of a Surgical Robotic Manipulator,” filed on March 15, 2014, the contents of which are incorporated herein by reference. In addition to, or instead of, manipulator 14 and tool 20, separate handheld surgical tools may be used.

[0043] Tool 20 includes an energy applicator 24 designed to contact the tissue of the patient 12 at the target site. In one embodiment, the energy applicator 24 is a bur (cutting instrument) 25. The bur 25 may be spherical and include a center, radius (r), and diameter of the sphere. Alternatively, the energy applicator 24 may be a drill bit, a saw blade 27 (see alternative tool in Figure 1), or an ultrasonic vibrating tip, etc. Tool 20 and / or energy applicator 24 (i.e., tool 20 or energy applicator 24 or both) may include any geometric features, such as perimeter, circumference, radius, diameter, width, length, volume, area, surface / plane, operating envelope range (along any one or more axes), etc. How the tool 20 is positioned relative to the tissue at the target site to perform the desired treatment may be determined by considering the geometric features. In some of the embodiments described herein, for convenience and ease of illustration, a spherical bar having a tool center point (TCP) and a sagittal saw blade having a TCP are described, but there is no intention to limit the tool 20 to any particular form.

[0044] The tool 20 may include a tool controller 21 that controls the operation of the tool 20, such as controlling power to the tool 20 (e.g., to the tool drive such as the rotary motor of the tool 20), controlling the movement of the tool 20, and / or controlling the irrigation / suction of the tool 20 (i.e., controlling power to the tool 20 (e.g., to the tool drive such as the rotary motor of the tool 20), controlling the movement of the tool 20, or controlling the irrigation / suction of the tool 20, or all of these). The tool controller 21 may communicate with the manipulator controller 26 or other components. The tool 20 may also include a user interface UI with one or more displays and / or input devices (e.g., push buttons, triggers, sensors, switches, keyboards, mice, microphones (voice control), gesture control devices, touchscreens, joysticks, foot pedals, etc.) (i.e., one or more displays or input devices or both), which is connected to the tool controller 21, the manipulator controller 26, and / or other controllers described herein (i.e., the tool controller 21, the manipulator controller 26, or other controllers described herein, or all of them). The manipulator controller 26 controls the state of the tool 20 (e.g., TCP) with respect to a coordinate system such as the manipulator coordinate system MNPL (e.g., position and / or orientation (i.e., position or orientation or both)). The manipulator controller 26 may also control velocity (linear velocity or angular velocity), acceleration, or other derivatives of the tool 20's motion.

[0045] In one embodiment, the tool center point (TCP) is a predetermined reference point defined on the energy applicator 24. The TCP has a pose that is known or computable (i.e., not necessarily static) with respect to other coordinate systems. The geometric shape of the energy applicator 24 is known in the TCP coordinate system or defined with respect to the TCP coordinate system. The TCP may be located at the spherical center of the bar 25 of the tool 20, or at the distal end of the saw blade 27, so that only one point is tracked. The TCP may be defined in various ways depending on the configuration of the energy applicator 24. The manipulator 14 may be able to determine the pose of the TCP using a joint / motor encoder or any other non-encoder position sensing method. The manipulator 14 may be able to determine the TCP pose using joint measurements and / or use a technique to directly measure the TCP pose (i.e., it may be able to determine the TCP pose using joint measurements, or use a technique to directly measure the TCP pose, or both). Control of the tool 20 is not limited to the center point. For example, the tool 20 can be represented using any suitable primitive, mesh, etc.

[0046] System 10 further includes a navigation system 32. One embodiment of the navigation system 32 is described in U.S. Patent No. 9,008,757, filed September 24, 2013, entitled “Navigation System Including Optical and Non-Optical Sensors,” which is incorporated herein by reference. The navigation system 32 tracks the movement of various objects. Such objects include, for example, a manipulator 14, a tool 20, and anatomical structures, such as the femur F, pelvis PEL, and tibia T. The navigation system 32 tracks these objects and collects state information of the objects with respect to the (navigation) localizer coordinate system LCLZ. The coordinates of the localizer coordinate system LCLZ can be transformed using transformations to the manipulator coordinate system MNPL, to other coordinate systems, and / or vice versa (i.e., to the manipulator coordinate system MNPL, to other coordinate systems, or vice versa, or all of them).

[0047] The navigation system 32 may include a trolley assembly 34 that houses a navigation controller 36 and / or other types of control units (i.e., the navigation controller 36, or other types of control units, or both). The navigation user interface UI communicates operablely with the navigation controller 36. The navigation user interface includes one or more displays 38. The navigation system 32 may use one or more displays 38 to display a graphical representation of the relative state of a tracked object to the user. The navigation user interface UI further includes one or more input devices for inputting information into the navigation controller 36 or for selecting / controlling a particular aspect of the navigation controller 36. Such input devices include interactive touchscreen displays. However, the input devices may include one or more of the following: push buttons, keyboards, mice, microphones (voice control), gesture control devices, and foot pedals.

[0048] The navigation system 32 also includes a navigation localizer 44 connected to a navigation controller 36. In one embodiment, the localizer 44 is an optical localizer and includes a camera unit 46. The camera unit 46 has an outer housing 48 that houses one or more optical sensors 50. The localizer 44 may also include its own localizer controller 49 and may further include a video camera VC.

[0049] The navigation system 32 includes one or more trackers. In one embodiment, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker 54, a second patient tracker 55, and a third patient tracker 56. In the embodiment shown in Figure 1, the manipulator tracker (i.e., tracker 52A) is securely attached to the tool 20, the first patient tracker 54 is securely fixed to the femur F of patient 12, the second patient tracker 55 is securely fixed to the pelvis PEL of patient 12, and the third patient tracker 56 is securely fixed to the tibia T of patient 12. In this embodiment, the patient trackers 54, 55, and 56 are securely fixed to the bone. The pointer tracker PT is securely fixed to a pointer P, which is used to align anatomical structures to the localizer coordinate system LCLZ. The manipulator trackers 52A and 52B can be fixed to any suitable component of the manipulator 14, in addition to or separate from the tool 20, for example, to the base 16 (i.e., tracker 52B), or to any one or more links 18 of the manipulator 14. The trackers 52A, 52B, 54, 55, 56, and PT can be fixed to their respective components in any suitable way. For example, the trackers may be rigidly fixed, flexibly connected (by optical fiber), or physically separated (e.g., by ultrasound), as long as there is a suitable (supplementary) way to identify the relationship (measurement) between each tracker and the object it is associated with.

[0050] Any one or more of the trackers may include an active marker 58. The active marker 58 may include a light-emitting diode (LED). Alternatively, trackers 52A, 52B, 54, 55, 56, and PT may have passive markers, such as reflectors that reflect light emitted from the camera unit 46. Other suitable markers not specifically described herein may be used.

[0051] The localizer 44 tracks the trackers 52A, 52B, 54, 55, 56, and PT to determine their states, which correspond to the states of the objects to which each tracker 52A, 52B, 54, 55, 56, and PT is attached. The localizer 44 may perform known triangulation techniques to determine the states of the trackers 52, 54, 55, 56, PT, and their associated objects. The localizer 44 provides the states of the trackers 52A, 52B, 54, 55, 56, and PT to the navigation controller 36. In one embodiment, the navigation controller 36 determines the states of the trackers 52A, 52B, 54, 55, 56, and PT and transmits them to the manipulator controller 26. The object states used herein include, but are not limited to, data defining the position and / or orientation of the tracked object (i.e., the position or orientation of the tracked object or both), or equivalents / derivatives of the position and / or orientation (i.e., the position or orientation or both). For example, a state could be the object's pose and may include linear velocity data and / or angular velocity data (i.e., linear velocity data or angular velocity data or both), etc.

[0052] The navigation controller 36 may comprise one or more computers or any other suitable form of controller. The navigation controller 36 comprises a central processing unit (CPU) and / or other processors (i.e., a central processing unit (CPU) or other processors or both), memory (not shown), and storage (not shown). The processors may be any type of processor, microprocessor, or multiprocessor system. Software is loaded into the navigation controller 36. The software, for example, translates signals received from the localizer 44 into data representing the position and orientation of the tracked object. The navigation controller 36 may optionally or alternatively comprise one or more microcontrollers, field-programmable gate arrays, system-on-chip, discrete circuitry, and / or other suitable hardware capable of performing the functions described herein (i.e., one or more microcontrollers, field-programmable gate arrays, system-on-chip, discrete circuitry, or other suitable hardware capable of performing the functions described herein, or all of them), software, or firmware. The term processor is not intended to limit any embodiment to a single processor.

[0053] One embodiment of the navigation system 32 is shown that uses triangulation techniques to determine the state of an object, but the navigation system 32 may have any other suitable configuration for tracking the manipulator 14, the tool 20, and / or the patient 12 (i.e., any other suitable configuration for tracking the manipulator 14, the tool 20, or the patient 12, or all of them). In another embodiment, the navigation system 32 and / or the localizer 44 (i.e., the navigation system 32 or the localizer 44 or both) are ultrasound-based. For example, the navigation system 32 may include an ultrasound imaging device connected to a navigation controller 36. The ultrasound imaging device images any of the aforementioned objects, e.g., the manipulator 14, the tool 20, and / or the patient 12 (i.e., the manipulator 14, the tool 20, or the patient 12, or all of them), and generates a state signal to the navigation controller 36 based on the ultrasound image. The ultrasound image may be 2D, 3D, or a combination of both. The navigation controller 36 can process the image in near real-time and determine the state of the object. The ultrasonic imaging device can have any suitable configuration and may differ from the camera unit 46 shown in Figure 1.

[0054] In another embodiment, the navigation system 32 and / or localizer 44 (either the navigation system 32 or the localizer 44 or both) are radio frequency (RF) based. For example, the navigation system 32 may include an RF transceiver connected to a navigation controller 36. The manipulator 14, tool 20, and / or patient 12 (either the tool 20 or patient 12 or both) may be fitted with an RF emitter or transponder. The RF emitter or transponder may be passively or actively energized. The RF transceiver transmits an RF tracking signal and generates a status signal to the navigation controller 36 based on the RF signal received from the RF emitter. The navigation controller 36 may analyze the received RF signal and associate a relative state with the RF signal. The RF signal may be of any suitable frequency. The RF transceiver may be placed in any suitable location to effectively use the RF signal to track an object. Furthermore, the RF emitter or transponder may have any suitable structural configuration that differs significantly from the trackers 52A, 52B, 54, 55, 56, and PT shown in Figure 1.

[0055] In yet another embodiment, the navigation system 32 and / or localizer 44 (i.e., the navigation system 32 or the localizer 44 or both) are electromagnetically based. For example, the navigation system 32 may include an EM transceiver connected to a navigation controller 36. The manipulator 14, the tool 20, and / or patient 12 (i.e., the tool 20 or the patient 12 or both) may be fitted with any suitable EM components, such as magnetic trackers, electromagnetic trackers, or inductive trackers. The trackers may be passively or actively energized. The EM transceiver generates an EM field and, based on the EM signal received from the tracker, generates a state signal to the navigation controller 36. The navigation controller 36 may analyze the received EM signal to associate a relative state with the EM signal. Again, such embodiments of the navigation system 32 may have a different structural configuration from the configuration of the navigation system 32 shown in Figure 1.

[0056] The navigation system 32 may have any other suitable components or structures not specifically detailed herein. Furthermore, any of the techniques, methods, and / or components described herein (i.e., techniques, methods, or components, or all thereof) relating to the shown navigation system 32 can be implemented or provided in any other embodiment of the navigation system 32 described herein. For example, the navigation system 32 may utilize inertial tracking alone or any combination of tracking techniques, and may additionally or alternatively include fiber optic-based tracking and machine vision tracking, etc. In particular embodiments of the present invention, optical IR tracking is utilized, but the concepts and techniques described herein may be used in conjunction with any sufficiently accurate 6D tracking technique. Furthermore, it is assumed that existing surgical navigation systems already include a suitable method for aligning preoperative image data and surgical plans with the patient before surgery.

[0057] System 10 includes a control system 60, which comprises any one or more of the manipulator controller 26, navigation controller 36, and tool controller 21 among a number of components. The control system 60 includes one or more software programs and software modules. A software module may be part of one or more programs that operate in the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof, to process data and assist in the control of System 10. The software programs and / or modules (i.e., software programs or modules or both) include computer-readable instructions, which are stored in non-temporary memory 64 of the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof, and are executed by one or more processors 70 of the controllers 21, 26, and 36. Memory 64 can be any suitable configuration of memory, such as RAM or non-volatile memory, and can be executed locally or from a remote database. Furthermore, a software module for prompting and / or communicating with the user (i.e., prompting, communication, or both) may form part of one or more programs and may include instructions stored in memory 64 of the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof. The user may communicate with the software module by interacting with any of the input devices of the navigation user interface UI or other user interface UIs. The user interface software may run on a device separate from the manipulator controller 26, navigation controller 36, and / or tool controller 21 (i.e., the manipulator controller 26, navigation controller 36, or tool controller 21, or all of them).

[0058] The control system 60 may comprise input devices, output devices, and processing devices in any suitable configuration appropriate for performing the functions and methods described herein. The control system 60 may comprise a manipulator controller 26, a navigation controller 36, or a tool controller 21, or any combination thereof, or only one of these controllers. These controllers may communicate via a wired bus or communication network, via wireless communication, or otherwise. The control system 60 may also be referred to as a controller. The control system 60 may comprise one or more microcontrollers, field-programmable gate arrays, system-on-chips, discrete circuits, sensors, displays, user interfaces, indicators, and / or other suitable hardware capable of performing the functions described herein (i.e., one or more microcontrollers, field-programmable gate arrays, system-on-chips, discrete circuits, sensors, displays, user interfaces, indicators, or other suitable hardware capable of performing the functions described herein, or all of them), software, or firmware.

[0059] [II. Spatial Recognition Display Techniques] This specification describes systems, methods, and techniques relating to spatially aware displays for computer-assisted intervention. A new paradigm of display and visual interaction is presented, aimed at reducing the complexity of understanding spatial transformations between the user's (e.g., surgeon's) viewpoint, physical objects (e.g., patient), 2D and 3D data (e.g., pre- and intra-operative patient data), and tools during computer-assisted intervention. For example, an intervention display in a surgical navigation system can be aligned to both the patient and the surgeon's view. This technique allows the surgeon to maintain their direct view of the patient without requiring any additional displays or direct view enhancements. In some embodiments, monitors used in the operating room are aligned to both the patient and the surgeon's viewpoint. This allows the physician to easily associate their view of the tools and the patient with a virtual representation of patient data. The surgeon's direct view of the patient and the surgeon's workspace can remain unchanged. The position and orientation of the display are integral parts of the visualization pipeline. Therefore, the display's pose is dynamically tracked relative to other objects, such as the patient, instruments, and, in some embodiments, the surgeon's head. This information is then used as input to an image-guided surgery visualization user interface.

[0060] At least two embodiments are presented. The first embodiment uses a “Fixed View Frustum” that associates the display pose with the patient and the tool. In the Fixed View Frustum technique, the display is tracked, for example, by attaching a tracking marker and calibrating the spatial relationship between the physical display and the tracking marker.

[0061] The second embodiment is built upon the mirror metaphor and extends the first technique by integrating the surgeon's head pose as an additional parameter in the visualization pipeline, in which case the display is associated with a "Dynamic Mirror View Frustum". In the Dynamic Mirror View Frustum visualization technique, the surgeon's viewpoint is tracked. Estimating the surgeon's viewpoint can be achieved by using a head tracking target or by attaching a camera to the surgical display and combining it with existing video-based head pose estimation algorithms. Once tracking information in a common global coordinate system is available, the spatial relationships between the patient, surgical display, tools, and surgeon's viewpoint can be calculated by deriving relevant transformations from the spatial relationships of the tracked entities.

[0062] These new display and visual analysis paradigms aim to reduce the complexity of understanding spatial transformations between the user's viewpoint, physical objects (O), pre- and intra-operative 2D and 3D data, and surgical tools 110, 20 during computer-assisted interventions, while minimizing changes to the current setup. The display, tools, and user's head can be tracked using any surgical tracking system compatible with integration into computer-assisted intervention systems. The proposed solutions allow physicians to easily correlate their views of the tools and patient with virtual data on the surgical monitor. Users can interact with patient data simply by intuitively moving their viewing position to observe patient data from different perspectives relative to the patient's position, without the need for interaction devices such as a mouse or joystick.

[0063] [A. Fixed-view frustum visualization technique] Referring to Figures 2–8, one exemplary visualization method available in the surgical system 10 includes a Fixed View Frustum (FVF) to assist user interaction with physical objects (O). Physical objects (O) can be physical anatomical structures, as shown in the figures, or any other objects requiring interaction, setup, or intervention, such as surgical devices, robotic devices, and surgical training models of anatomical structures. For brevity, physical anatomical structures are shown as objects, but the concept is not limited to these.

[0064] One or more external screens or display devices 100 are provided. These display devices 100 may be displays 38 on the navigation trolley 34 assembly, or any other external displays that are separate from the user and not mounted by the user. The display devices 100 may be any suitable type of display, including but not limited to LED, LCD, OLED, touchscreen, and holographic displays, and may display any type of image. The display devices 100 may also have any suitable geometric shape, including rectangular, square, or circular.

[0065] During use, the display device(s) 100 is positioned on the side (S1) of the physical object(O) opposite to the side (S2) where the user is located. In other words, the physical object(O) is located between the user's viewpoint(V) and the display device(s) 100. Thus, the display device(s) 100 used herein are distinct from a head-mounted display or tablet screen located between the user's viewpoint(V) and the object(O). Here, the physical object(O) is displayed as a virtual representation for illustrative purposes only.

[0066] Of course, the user is not prohibited from moving between the physical object (O) and the display device 100. However, embodiments of spatial awareness displays take into account the practical reality that a surgeon would want to visualize the physical object (O) on the side they are currently on, and that an inverted perspective is usually provided on the side of the physical object (O) opposite to the surgeon. This will be further understood in the context of the field of view of a virtual camera facing the display device 100 in front, as described below.

[0067] The display device 100 defines a plane (P). The plane (P) is defined to be parallel to or coincide with the actual front of the display device 100. Here, the plane (P) is a virtual object used for computational purposes, as described below. The control system 60, including a navigation system 32 with any one or more controllers described herein, is configured to align computer images (R) to physical objects (O). Alignment of physical objects (O) can be performed by any suitable method, not limited to any technique, including digitization of the physical object (O), touchless registration (e.g., ultrasound), or 2D / 3D image registration using an imaging device (e.g., CT, X-ray). The computer images (R) can be represented as virtual models (VMs) of any part of the physical object (O). These computer images (R) can be rendered on the display device 100. These computer images (R) may be static or dynamic and may include actual / real videographic images, mixed reality, augmented reality, virtual reality images or models, or any combination thereof.

[0068] The control system 60 tracks the pose of the physical object (O) via the localizer 44 using any of the aforementioned patient tracking techniques, including but not limited to the patient trackers 54, 55, and 56. In some embodiments, the localizer 44 may also track the pose of the display device 100 in a coordinate system common to the physical object (O). The display device 100 may be tracked using the display tracker 102 or by any other suitable technique as described above. For example, the display device 100 may be tracked using machine vision (e.g., with or without a tracker), or using optical or non-optical techniques. Alternatively or additionally, when an electric adjustment system is provided, as described later, the pose of the display device 100 may be determined based on kinematic data independently of the localizer 44.

[0069] The control system 60 controls the display device 100 to render a aligned computer image (R) according to the tracked poses of at least the physical object (O) and the display device 100. Thus, the rendering of the computer image (R) depends at least on the pose of the physical object (O) and the pose of the display device 100. Specifically, referring to Figure 2, the rendering perspective is based on the viewpoint or field of view (FOV) of a virtual camera (VC) facing the plane (P) of the display device 100. The field of view (FOV) may be based on a pinhole camera model projected toward the display device 100. The field of view (FOV) is also known as the frustum (F) and is described below.

[0070] The virtual camera (VC) is positioned at a virtual location (VP) on the side (S1) of the physical object (O) opposite to the side (S2) of the display device 100. In Figure 2, since the virtual camera (VC) is positioned in front of the plane (P) of the display device 100, the computer image (R) displays the side of the physical object (O) on the side (S1), rather than a mirror view of the side (S2).

[0071] The virtual position (VP) can be located at a predetermined distance (d) from the plane (P) of the display device 100. The distance (d) is based on the Z-axis line drawn between the virtual position (VP) of the virtual camera (VC) and the plane (P). The Z-axis can be defined with respect to the virtual position (VP) or the display device 100. The virtual position (VP) can be any location on the virtual camera (VC) and can be defined by a single point, multiple points, or the geometric shape of any kind of surface or volume. In some embodiments, the virtual position (VP) is kept fixed at a predetermined distance (d). Alternatively, the predetermined distance (d) can be changed depending on specific conditions. For example, the user can specify a preference for the distance (d) to the control system 60. The distance (d) may be automatically changed depending on conditions such as, but not limited to, the type of surgical procedure, specific steps of the procedure, the tracked pose of the user's viewpoint (V), the tracked pose of an object, the tracked pose of the display device 100, or the tracked poses of tools 110, 20. In one embodiment, the distance (d) may be defined within any suitable distance and any suitable range, including but not limited to any distance between 1 meter and 5 meters, or any distance within these ranges.

[0072] In one embodiment, the virtual position (VP) of the virtual camera (VC) traverses the plane (P) of the display device 100. In other words, the Z-axis line drawn between the virtual position (VP) and the plane (P) of the display device 100 defines an angle with respect to the plane (P), and the angle is in the range of 0 to 180 degrees. In a more specific embodiment, the virtual position (VP) of the virtual camera (VC) is orthogonal to the plane (P) of the display device 100. In other words, the line drawn between the virtual position (VP) and the plane (P) of the display device 100 is perpendicular to the plane (P) at 90 degrees. In other words, in this embodiment, the computer image (R) is rendered in perspective projection on an axis orthogonal to the display device 100 from a given distance (d).

[0073] The virtual position (VP) of the virtual camera (VC) also has X and Y coordinate positions with respect to the plane (P) of the display device 100. The X and Y coordinates may be defined with respect to the virtual position (VP) or the display device 100. In one embodiment, the X and Y coordinates are defined on a plane parallel to the plane (P) of the display device 100. However, if, for example, the virtual camera (VC) is projecting toward the display device 100 at a non-orthogonal angle, the X and Y coordinate plane does not need to be parallel to the plane (P).

[0074] Furthermore, regardless of whether the projection crosses or orthogonals the display device 100, the line drawn between the virtual position (VP) and the plane (P) may be oriented towards the geometric center of the display device 100. Alternatively, the line drawn between the virtual position (VP) and the plane (P) may be offset from the geometric center of the display device 100. For example, the offset position may be near a corner or edge of the display device 100. The XY arrangement of the virtual position (VP) relative to the display device 100 may be changed depending on certain conditions. For example, the user can specify their preference for the XY arrangement of the virtual position (VP) to the control system 60. The XY arrangement of the virtual position (VP) may be automatically changed depending on conditions such as, but are not limited to, the type of surgical procedure, a specific step of the procedure, the tracked pose of the user's viewpoint (V), the tracked pose of an object, the tracked pose of the display device 100, or the tracked pose of the surgical tool 20. In one embodiment, distance (d) may be defined within any suitable distance and any suitable range, including but not limited to any distance between 1 meter and 5 meters, or any distance contained within these ranges.

[0075] The rendering of a computer image (R) based on a virtual camera (VC) is performed using a viewing frustum (F), which originates from the virtual camera (VC) and passes through an object, in this case the plane (P) of the display device 100. The viewing frustum (F) can be the spatial region of a 3D modeled world that can be displayed on the display device 100 and can be considered the field of view of the virtual camera (VC). The vertex of the viewing frustum (F) is the zero point from which the virtual camera (VC) originates and can also be considered the virtual position (VP) of the virtual camera (VC). In one embodiment, the plane (P) of the display device 100 may be located at the base (B) of the viewing frustum (F), i.e., the far clipping plane (FCP). The viewing frustum (F) may have a near-clipping plane (NCP) that is close to the virtual camera (VC). The far-clipping plane (FCP) and near-clipping plane (NCP) virtually cut the viewing frustum (F) perpendicular to the field of view, so that objects closer to the camera than the near-clipping plane (NCP) or beyond the far-clipping plane (FCP) are not rendered in the computer image (R). Objects that are partially or completely outside the viewing frustum (F) may be excluded from the rendering process to improve processing efficiency. The computer image (R) on the display device 100 is projected based on the viewing frustum (F). The viewing frustum (F) can be based on any plane section or any suitable three-dimensional shape, including a pyramid or a cone, etc. The viewing frustum (F) may have any configuration other than the configuration shown in the figure or the configuration described herein.

[0076] In the FVF technique, the virtual position (VP) of the virtual camera (VC) is automatically updated by the control system 60 in response to (manual or motorized) adjustments to the pose of the display device 100. When the user moves or rotates the display device 100, the position of the virtual camera (VC) is automatically updated. Thus, in contrast to standard visualization displays in surgical navigation systems, the intelligent mapping of real-world objects to their images on the screen is simplified.

[0077] Examples of this type of fixed-view frustum display visualization are shown in Figures 3A and 3C. A real-world example is shown in Figure 3A, and two simulations are shown in Figures 3B and 3C, respectively. Figure 3A shows an operator manually moving the display device 100 to make the internal virtual model (VM) of a physical object (O) visible. As the user adjusts the display device 100, the computer rendering (R) on the display device 100 updates accordingly. Figures 3B and 3C provide simulation diagrams of the FVF technique based on two different viewpoints of a virtual camera (VC) with the same configuration. As shown in comparison, the display device 100 changes its pose between Figure 3B and Figure 3C, and the virtual position (VP) of the virtual camera (VC) updates accordingly. The computer image (R) also changes its perspective according to the relative position between the viewing frustum (F) and the physical object (O).

[0078] [i. Screen Parallel Slice Visualization (SPSV)] Referring to Figures 4-6, the screen parallel slice visualization (SPSV), a sub-technique of the FVF method, is explained. While FVF visualization is intuitive and can display 3D data, the technique can be further implemented using slice visualization. Therefore, slice views can be incorporated into the spatial awareness visualization concept described.

[0079] Referring to Figure 4A, in one embodiment, this can be achieved by slicing a 3D virtual model (VM) of a physical object (O) into multiple slices (SL1, SL2, SL3, ..., SLN). In one embodiment, the slices (SL) are made in planes parallel to the plane (P) of the display device 100. The virtual model (VM) may be a CT model, an X-ray model, an MRI model, or a model created using any other type of imaging technique. Alternatively, instead of slicing the 3D virtual model (VM), the imaging data of the physical object (O) may include multiple slices that can be obtained from computer memory, and these multiple slices may or may not be combined into a 3D model.

[0080] In one embodiment, referring to Figure 4B, the display slice (SL) may be based on a designated plane (p) sliced ​​through a viewing frustum (F). The designated plane (p) may be fixed relative to the position of the virtual camera (VC), or it may be dynamically changed based on any of the conditions described herein. When the field of view of the virtual camera (VC) passes through a physical object (O), the control system 60 may immediately acquire a slice (SL) corresponding to the intersection of the designated plane (p) and the physical object (O).

[0081] The orientation of the computer image (R) of the slice (SL) can be adjusted based on the pose of the display device 100. As shown by comparing Figures 5A and 5B, the user is given an interaction method in which the display device 100 is rotated, and then the virtual models (T) of the slice (SL) and tools 110 and 20 are also rotated. In Figures 5A and 5B, the physical object (O) and tools 110 and 20 are in the same relative position.

[0082] Alternatively or additionally, the orientation of the slice (SL) may be set based on tools 110, 20 brought into the frustum (F) of a virtual camera (VC), as shown in Figure 6. The poses of tools 110, 20 may be tracked by a localizer 44 using any suitable means as described herein. The rendered computer image (R) may include images of tools 110, 20 and slices (SL) using the fixed-view frustum method discussed above. Alternatively or additionally, the orientation of the slice (SL) may be set based on the user's viewpoint defined by the tracked pose of an external camera (C) facing the user or a viewpoint tracking system (VTS), such as a head-mounted device 120 (described in detail below).

[0083] During the use of these techniques, the display slice (SL) can be dynamically changed to another slice (SL). The slice (SL) can be changed depending on the pose of the physical object (O), the display device 100, or any combination thereof. Additionally or alternatively, the slice (SL) can be changed using any of the input devices described herein. The slice (SL) can also be changed using the tools 110, 20, as shown in Figure 6. For example, the control system 60 may automatically change the slice (SL) based on the pose, position, and / or distance (i.e., pose, position, or distance, or all of them) or orientation of the tools 110, 20 relative to either the physical object (O) or the display device 100. In some embodiments, as shown in Figure 4B, a portion of the slice (SL) may be displayed based on a (2D or 3D) virtual boundary (VB) associated with the tip 114 of the tools 110, 20. For example, the virtual boundary (VB) can be defined by any shape (e.g., rectangle, box, circle, or sphere) with any appropriate dimensions (e.g., a diameter of 200 mm), and the tool tip 114 is located at the center of the boundary (VB). When the tools 110, 20 are brought close to the object (O) so that the object (O) intersects the virtual boundary (VB), the control system 60 can dynamically present or modify the display slice (SL) corresponding to the intersection. Depending on the position of the virtual boundary (VB), the slice (SL) can be presented in its entirety (as shown in Figure 4B) or trimmed (Figure 6).

[0084] [ii. Contextually accurate rendering] Referring to Figure 7, the control system 60 is configured to render a computer image (R) in a precise context. In other words, the computer image (R) rendering takes into account the relative spatial layering of the object (O), tools 110, 20, and / or slices (SL) (i.e., the object (O), tools 110, 20, or slices (SL), or all of them) in a common coordinate system, thereby simulating their actual spatial positions in the real coordinate system in which these items exist. As shown in Figure 7, the tools 110, 20 are physically inserted into the object (O), and the display device 100 presents virtual models (T) of the tools 110, 20 and their corresponding slices (SL) to the 3D model (VM) of the object (O). However, as shown, these renderings (R) are layered based on the viewpoint of a virtual camera (VC). In other words, the virtual models (T) of the tools 110, 20 are deliberately obscured by the corresponding portions of the 3D model (VM) of the object (O). In this case, the thoracic structure of the rib cage obstructs the shafts of tools 110 and 20. Similarly, the slice (SL) is displayed such that a portion of the slice (SL) is obstructed by a corresponding portion of the 3D model (VM) of the object (O). In this case, the slice (SL) is partially obstructed by several ribs. Although this technique partially obstructs the visualization, it is beneficial to the user to visualize the environment in a context that closely resembles the real-world spatial arrangement of the object (O), slice (SL), and tools 110 and 20. This visualization technique can be used in any of the techniques described herein, including the FVF technique and the SPSV technique, and in any of the specific embodiments, conditions, and / or situations (i.e., embodiments, conditions, or situations, or all of them) described herein.

[0085] [iii. Adjustment System] As shown in Figure 8, the display device 100 may optionally be connected to an adjustment system 104. The adjustment system 104 can be passive or active and may have any features or possible configurations of the robot manipulator 14 described above. For example, the adjustment system 104 may comprise a number of links (L), joints (J), and actuators (M) that drive the joints (J). The adjustment system 104 may be controlled manually or automatically to adjust the pose of the display device 100. The pose of the display device 100 may be moved with up to six degrees of freedom (three translational degrees of freedom and three rotational degrees of freedom). A sensor (S) is provided on one of the components of the adjustment system 104 to detect the movement of the adjustment system 104. The control system 60 may use the measurements from the sensor (S) to kinematically derive the pose of the display device 100. This may be done in addition to, or instead of, using the localizer 44. The sensor (S) can be any suitable configuration, including but not limited to a motor current sensor, a joint position sensor or encoder (such as a rotary encoder, absolute encoder, incremental encoder, or virtual absolute encoder), an optical sensor, or an inertial sensor (such as an accelerometer, inclinometer, or gyroscope). Any of the sensors (S) can also be provided on the display device 100 itself.

[0086] In some cases, as shown in Figure 8, one or more user input devices 106 may be connected to the control system 60 by wire or wirelessly, thereby enabling the user to control the pose of the display device 100. The input devices 106 include, but are not limited to, a foot pedal 106a, a handheld pendant 106b, a display input 106c such as a tablet, smartphone, or navigation system display, tools 110, 20, and / or an eye-tracking system (VTS) such as an external camera (C) or head-mounted device 120 facing the user (i.e., a foot pedal 106a, a handheld pendant 106b, a display input 106c such as a tablet, smartphone, or navigation system display, tools 110, 20, or an eye-tracking system (VTS) such as an external camera (C) or head-mounted device 120 facing the user, or all of them). The input device can be any of the aforementioned input devices, including but not limited to push buttons, sensors, switches, keyboards, mice, microphones (voice control), gesture control devices, touchscreens, joysticks, etc. The input device 106 receives commands from the user, and the control system 60 instructs one or more actuators M to adjust joint J according to the commands, thereby adjusting the pose of the display device 100.

[0087] The tracked tools 110 and 20 can also be used to trigger the control system 60 to control the adjustment system 104. In one embodiment, the pose of the tracked tools 110 and 20, whether it is position and / or orientation (i.e., position or orientation or both) or any derivative thereof (velocity, acceleration, etc.), can cause the control system 60 to adjust the pose of the display device 100. For example, the tracked pose of the tools 110 and 20 can be compared to the virtual position (VP) of the virtual camera (VC), the viewing frustum (F), the plane (P) of the display device 100, or any combination thereof. The control system 60 may evaluate this comparison with respect to a threshold condition or threshold measurement. If the control system 60 determines that the tools 110 and 20 have been moved in a manner that satisfies or exceeds a threshold condition, the control system 60 may command the adjustment system 104 to adjust the display device 100. This can be useful in a variety of situations, including, but not limited to, situations where tools 110 and 20 move closer to or away from an object (O), where tools 110 and 20 enter or exit a viewing frustum (F), or where tools 110 and 20 are kept within the view of a virtual camera (VC).

[0088] Continuing to refer to Figure 8, the head-mounted device 120 can also be used to trigger the control system 60 to control the adjustment system 104. In one embodiment, the pose of the head-mounted device 120 is tracked using any tracking technique or equivalent thereof as described herein. The tracked pose of the head-mounted device 120, whether it is position and / or orientation (i.e., position or orientation or both) or any derivative thereof (velocity, acceleration, etc.), can cause the control system 60 to adjust the pose of the display device 100. For example, the tracked pose of the head-mounted device 120 can be compared to the virtual position (VP) of the virtual camera (VC), the viewing frustum (F), the plane (P) of the display device 100, or any combination thereof. The control system 60 may evaluate this comparison with respect to a threshold condition or threshold measurement. If the control system 60 determines that the head-mounted device 120 has been moved in a manner that meets or exceeds a threshold condition, the control system 60 may command the adjustment system 104 to adjust the display device 100. This can be beneficial in a variety of situations, including, but not limited to, situations such as the head-mounted device 120 moving closer to or away from an object (O), the head-mounted device 120 entering or leaving a viewing frustum (F), or maintaining the head-mounted device 120 within the view of a virtual camera (VC).

[0089] Additionally or alternatively, any other viewpoint tracking system (VTS), such as an external camera (C) facing the user, can be used to track the user's viewpoint (V) and trigger the control system 60 to control the adjustment system 104.

[0090] The adjustment system 104 can be used in any of the techniques described herein, including the FVF technique and the SPSV technique, and in any of the specific embodiments, conditions, and / or circumstances described herein (i.e., embodiments, conditions, or circumstances, or all of them).

[0091] [B. Dynamic Mirror View Frustum Visualization Technique] Referring to Figures 9 to 12, another exemplary visualization method available in the surgical system 10 includes a dynamic mirror frustum (DMVF) to assist user interaction with physical objects (O).

[0092] As can be understood from the following description, the FVF technique and the DMVF technique have technical similarities. Therefore, all of the above descriptions concerning System 10, the FVF technique, and any physical, computational, and / or any technique associated therewith (i.e., physical, computational, or any technique, or all of them) are incorporated by reference and used together with and applicable to the DMVF technique described herein, and are therefore not repeated for the sake of brevity.

[0093] In the DMVF technique, as shown in Figure 9, the display device 100 defines a plane (P). The physical object (O) is positioned in front of the plane (P). In other words, the physical object (O) is positioned in front of the display screen of the display device 100.

[0094] The user viewing the display device 100 is also positioned in front of the plane (P). The user's viewpoint (V) is tracked using a viewpoint tracking system (VTS). In one embodiment, the viewpoint tracking system (VTS) includes a camera (C) facing the user. The viewpoint tracking system (VTS) may be the navigation system itself, part of the navigation system, or separate from the navigation system. The camera (C) may be a localizer 44 or a separate device. The camera (C) may be mounted on the display device 100 or elsewhere. The control system 60 may receive signals from the camera (C) and use a pose estimation algorithm to recognize changes in the position and / or orientation (i.e., position or orientation or both) of the user's face, eyes, head, or other features.

[0095] In another embodiment, the viewpoint tracking system (VTS) additionally or alternatively includes a head-mounted device 120 provided directly to the user. The head-mounted device 120 is positioned in front of the plane (P) and in front of the display screen of the display device 100. In one configuration, the head-mounted device 120 is positioned on a first side (S1) of the physical object (O), and the display device 100 is positioned on a second opposite side (S2) of the physical object.

[0096] The head-mounted device 120 has one or more trackable features (HTs) so that the pose of the head-mounted device 120 can be tracked by the navigation system 32 and / or the control system 60 (i.e., the navigation system 32 or the control system 60 or both). The trackable features (HTs) can be any of the aforementioned types or any equivalent thereof. In this technique, the head-mounted device 120 is any device configured to track the pose of the user's viewpoint, where pose means the position of the head-mounted device 120, and optionally position and orientation. The user's viewpoint can be defined by the user's general field of view, the orientation of the user's own head, and / or the user's line of sight (i.e., the user's general field of view, the orientation of the user's own head, or the user's line of sight, or all of them). The head-mounted device 120 may be, or may include, one or more trackers attached to any part of the user's head, and / or eyewear or headwear such as glasses, goggles, a headband helmet, or contact lenses (i.e., the head, eyewear or headwear, or all of them) (as shown, for example, in Figure 8). In one embodiment, the head-mounted device 120 is an optical see-through head-mounted display. Other embodiments of the head-mounted device 120 are also considered. In Figure 9, the viewpoint derived from tracking by the head-mounted device 120 is shown simply as a sphere, and the viewpoint faces the display device 100.

[0097] Similar to the FVF technique, the navigation system 32 and / or the control system 60 (i.e., the navigation system 32 or the control system 60 or both) align the computer image (R) to the physical object (O). The navigation system 32 and / or the control system 60 (i.e., the navigation system 32 or the control system 60 or both) track the poses of the physical object (O), the display device 100, and the user's viewpoint (V) relative to each other in a common coordinate system. The display device 100 is controlled to render the aligned computer image (R) according to the tracked poses of the physical object (O), the display device 100, and the user's viewpoint (V). In other words, the pose of each of these items can affect how the computer image (R) is displayed.

[0098] The perspective of the computer image (R) is based on the field of view, or frustum (F), of a virtual camera (VC) having a virtual position (VP) behind a plane (P) shown as the side (S3) in Figure 9, which is behind the display device 100. The virtual camera (VC) faces the back of the display device 100 and faces the user's viewpoint derived from the viewpoint tracking system (VTS). The physical object (O), the display device 100, and the user's viewpoint (V) are at least partially located within the frustum (F) in Figure 9. Because the virtual camera (VC) is positioned behind the plane (P) of the display device 100, the rendering (R) provides a visualization as if the display device 100 were a mirror image of the user's viewpoint. In Figure 9, the computer image (R) shows the side of the physical object (O) at the side (S2).

[0099] The virtual position (VP) of the virtual camera (VC) is automatically updated in accordance with adjustments to the tracked pose of the user's viewpoint (V). As the user moves left, right, forward, or backward, they can see correspondingly moving physical objects (O) and any tools 110, 20 on the mirror-like rendering (R) of the display device 100. By understanding the poses of the physical objects (O), the display device 100, and the user's viewpoint (V), the control system 60 can create a computer image (R) such that the computer image (R) follows the same rules as a real mirror.

[0100] Figures 10A to 10F illustrate examples of the DMVF technique. Specifically, simulated mirror views are shown in Figures 10A to 10D, and examples representing the real world are shown in Figures 10E to 10F. Comparing the third-person views shown in Figures 10A and 10C, the virtual position (VP) of the virtual camera (VC) is changed according to the user's viewpoint (V) (shown as a sphere) derived from the viewpoint tracking system (VTS). Figure 10B shows the first-person viewpoint as seen on the display device 100 in the scenario of Figure 10A. Figure 10C shows the first-person viewpoint as seen on the display device 100 in the scenario of Figure 10D.

[0101] The display device 100 displays objects in front of the display as if in a mirror, taking into account the screen, objects, and the user's viewpoint pose. This paradigm work can contribute to a more intuitive visualization of physical objects (O) by using motion parallax to observe structures from further desired viewpoints. Users can freely interact with data simply by looking, without necessarily requiring interaction devices such as a mouse or joystick. Since users do not need to constantly redefine their view of the object (O), they can switch interactivity on and off using any input device, such as a foot pedal. Users can change the visualization in a hands-free manner. Because the human natural visual system is adapted to looking in a real mirror, DMVF visualization provides an intuitive interface. DMVF also facilitates the exploration and demarcation of the optimal slice (SL) or rendering (R) for a given navigation task. The slice (SL) or rendering (R) can remain fixed during the operation until further data exploration is desired.

[0102] [i. Setting up a virtual camera and mirror frustum for DMVF] The virtual position (VP) of the virtual camera (VC) is dynamically changed using the DMVF method, at least in accordance with the tracked pose of the user's viewpoint (V). In one embodiment, the viewpoint of the virtual camera (VC) may be derived as described herein. In one embodiment, the control system 60 moves the virtual camera (VC) according to a projection matrix adjusted to fit a viewing frustum (F), so that the boundary feature (BF) of the viewing frustum (F) coincides with the feature (DF) of the display device 100. The feature (DF) of the display device 100 includes a point or other geometric shape that encodes the size, aspect ratio, position and orientation of the display device 100 in a common coordinate system or world coordinate system. In the embodiment of Figure 9, the viewing frustum (F) includes a truncated pyramid, the display device 100 is rectangular, the boundary feature (BF) of the viewing frustum (F) is a point on the side edge of the pyramid, and the feature (DF) is the fixed angle of the display device 100. Therefore, for any positional movement of the virtual camera (VC) in the coordinate system, the points on the edge of the viewing frustum (F) are made to coincide with the corners of the display device 100. Of course, this configuration can vary depending on the geometric shapes of the viewing frustum (F) and the display device 100. This allows for the acquisition of a distorted image (R) represented in the correct perspective as a specular reflection from the tracked pose of the user's viewpoint (V).

[0103] From the virtual position (VP) of the virtual camera (VC), the viewpoint position in the mirror world is p mirrored However, it can be calculated as follows. Define the perpendicular to the mirror plane (P) as (0;0;1). In the local coordinate system, mirroring is performed using matrix M flip This corresponds to multiplication of matrix M flip This is represented by [1] and takes into account the fixed features (DF) of the display device 100 (i.e., the four corners in this case).

number

[0104] This corresponds to scaling by -1 minute in the z direction. According to one embodiment, in order to provide a mirrored view, it is assumed that rendering depends on the position of the user's viewpoint (V) rather than the orientation of the user's viewpoint (V) relative to the mirror plane (P). A first transformation is performed to convert the world position of viewpoint p of the virtual camera (VC) to the local coordinates of the mirror. This rotates the viewing frustum (F) to the user's viewpoint. Next, matrix M flip Mirroring is then performed, which provides a perspective projection onto the display device 100. Finally, the mirror's mirror local coordinates are converted back to the world position of the virtual camera's (VC) viewpoint p, thereby converting the user's viewpoint (V) to the vertex of the viewing frustum (F). The result is a projection matrix represented by the following equation:

number

[0105] In the equation shown in [2], the following annotation is used to represent the transformation from coordinate system A to coordinate system B.

number

[0106] In one embodiment, an off-axis projection matrix may be employed so that the computer image (R) on the display device 100 can render a mirrored view of a physical object (O) or tool 110, 20 in front of the display device 100 with respect to the user's pose. Here, off-axis means that the line drawn from the user's viewpoint (V) to the display device 100 passes through a position off-center, rather than the geometric center of the display device 100. In this situation, the frustum (F) becomes asymmetric, and its shape changes as the user's viewpoint (V) changes. The position of the user's viewpoint (V) relative to the geometric shape of the plane (P) of the display device 100 may be monitored by the control system 60. When the shape of the frustum (F) changes in accordance with the user's viewpoint (V), the boundary features (BF) of the frustum (F) change. In one embodiment, if the pose of the display device 100 and / or the user's viewpoint (i.e., the display device 100 or the user's viewpoint or both) may change, the viewing frustum (F) can be recalculated and / or updated (i.e., recalculated or updated or both) at frequent intervals, such as every frame.

[0107] The off-axis projection matrix has a mirrored viewpoint p at the vertices of the frustum (F) and base (B) that coincide with the surface or plane (P) of the display device 100. The frustum (F) is rotated to align with the user's viewpoint (V). This distorts the image to account for the viewpoint position, so that the image appears as if projected onto a plane tilted relative to the plane (P) of the display device 100. The effect of the modified user's viewpoint on the frustum (F), along with the distorted image, will be understood by comparing Figures 10A and 10B with Figures 10C and 10D. This embodiment relies on a graphics API to set up a projection matrix with a frustum (F) defined by near-clip plane (NCP) coordinates in view space, and then the frustum is rotated to be non-vertical and moved to align with the mirrored viewpoint. In other words, the base (B) of the viewing frustum (F) is dynamically changed to rotate outside the XY plane of the plane (P) of the display device 100 and positioned to correspond to the angle of the user's viewpoint (V) derived from the viewpoint tracking system (VTS).

[0108] The calculations provided above provide one embodiment of setting up a virtual camera (VC) for a DMVF. However, without deviating from the scope of the concept, there may be alternative methods for dynamically changing the virtual position (VP) of the virtual camera (VC) according to at least the user's viewpoint (V). For example, the base (B) of a viewing frustum (F) can be aligned with the display device 100. The boundary features (BF) of the viewing frustum (F) may exceed the geometric shape of the display device 100. In such cases, the control system 60 may restrict the boundary features (BF) of the viewing frustum (F) with respect to the geometric shape of the display device 100.

[0109] Figures 11A to 11C compare the relative positions of the virtual camera (VC) and frustum (F), the user's viewpoint (V), the display device 100, and the object (O) in various views of the X, Y, and Z coordinate systems, among three exemplary scenarios. In these embodiments, the position of the virtual camera (VC) and the user's viewpoint (V) may be mirrored separately with respect to the plane (P) of the display device 100 for each of the X, Y, and Z axes. For example, the virtual camera (VC) and the user's viewpoint (V) are substantially equal in distance from the display device 100 in each of the X, Y, and Z directions. In visualizations from the XY plane, as a result of this mirroring effect, the position of the virtual camera (VC) and the user's viewpoint (V) appear to coincide. However, perfect mirroring is not required in all embodiments, and the distances from (VC) and (V) to the display device 100 may be uneven for each axis, and such distances can be customized or modified.

[0110] In Figure 11A, the user's viewpoint (V) is on the axis at a given position relative to the display device 100. The base (B) of the viewing frustum (F) of the virtual camera (VC) is aligned with the XY plane of the display device 100. A computer image (R), positioned in the center and aligned with the user's central viewpoint, is rendered by projection.

[0111] In Figure 11B, the user's viewpoint (V) moves to the left of the center (from the user's perspective), further away from the display device 100 than its given position in Figure 11A. This relative change in the movement of the user's viewpoint (V) causes the virtual camera (VC) to change accordingly. In other words, the virtual camera (VC) moves away from the display device 100 (from the virtual camera's perspective). This correctly positions the virtual camera (VC) relative to the user's viewpoint (V). The viewing frustum (F) is rotated outside the XY plane of the display device 100 (counterclockwise when viewed from above) to align with the user's viewpoint (V). In some cases, the viewing frustum (F) changes in response to the virtual camera (VC) moving away from the display device 100, causing the far clipping plane (FCP) to move away from the vertex of the viewing frustum (F). The computer image (R), rotated to align with the user's off-center viewpoint, is rendered by projection. As the user's viewpoint (V) moves away from the display device 100, the computer image (R) also renders objects such as physical objects (O) within the viewing frustum (F) to appear smaller in size compared to Figure 11A.

[0112] In Figure 11C, the user's viewpoint (V) moves to the right (from the user's perspective) and is closer to the display device 100 than in the given position in Figure 11A. This relative change in the movement of the user's viewpoint (V) causes the virtual camera (VC) to move to the left (from the virtual camera's perspective) and is closer to the display device 100. This correctly positions the virtual camera (VC) relative to the user's viewpoint (V). The viewing frustum (F) is rotated outside the XY plane of the display device 100 (clockwise when viewed from above) to align with the user's viewpoint (V). In some cases, the viewing frustum (F) changes as the position of the virtual camera (VC) moves away from the display device 100, causing the far clipping plane (FCP) to move closer to the vertex of the viewing frustum (F). The computer image (R), rotated to align with the viewpoint off-center from the user, is rendered by projection. As the user's viewpoint moves closer to the display device 100, the computer image (R) also renders objects such as physical objects (O) within the viewing frustum (F) to be larger in size compared to Figure 11A.

[0113] The dimensions of the components in Figure 11 are provided for illustrative purposes only and may not be to an exact scale. Furthermore, while the XZ plane is shown to illustrate horizontal movement between the user's viewpoint and the virtual camera (VC), the principles described herein can be fully applied to vertical movement between the user's viewpoint and the virtual camera (VC) in the XY plane.

[0114] [ii. Visualization of viewpoint-facing slices] Here, referring to Figures 12A and 12B, Viewpoint Facing Slice Visualization (VFSV), a sub-technique of the DMVF method, is explained. While DMVF visualization is intuitive and can display 3D data, the intention is to further incorporate slice views into the spatial awareness DMVF visualization concept described.

[0115] Data slicing can be performed according to any of the techniques previously described with respect to the SPSV technique and as shown with respect to Figures 4A and 4B, and therefore, for the sake of brevity, will not be repeated in this specification. Furthermore, the selection of display slices (SLs) can be performed according to any of the techniques previously described with respect to the SPSV technique and as shown with respect to Figures 4A and 4B.

[0116] In one embodiment, the slice (SL) can be modified using the tools 110, 20. For example, the control system 60 may automatically modify the slice (SL) based on the pose, position, and / or distance (i.e., pose, position, or distance, or all of them) or orientation of the tools 110, 20 relative to either a physical object (O) or a display device 100. In some embodiments, as shown in Figure 4B, a portion of the slice (SL) may be displayed based on a (2D or 3D) virtual boundary (VB) associated with the tip 114 of the tools 110, 20. For example, the virtual boundary (VB) can be defined by any shape (e.g., rectangle, box, circle, or sphere) with any suitable dimensions (e.g., a diameter of 200 mm), with the tool tip 114 at the center of the boundary (VB). When the tools 110, 20 are brought closer to the object (O) so that the object (O) intersects with the virtual boundary (VB), the control system 60 may dynamically present or modify the display slice (SL) corresponding to the intersection. A slice (SL) can be presented as a whole (as shown in Figure 4B) or trimmed (Figure 6), depending on the position of the virtual boundary (VB).

[0117] Additionally or alternatively, the slice (SL) may be modified depending on the pose of a physical object (O), the display device 100, the user's viewpoint, or any combination thereof. The slice (SL) may also be modified using any of the input devices 106 described herein.

[0118] The orientation of the slice (SL) can be manipulated according to several different techniques. In the VFSV technique, the control system 60 tracks the display device 100 and the user's viewpoint (V) according to the DMVF technique. The position of the slice (SL) is based on the tracked positions of the tools 110, 20. However, the orientation of the slice (SL) is selected to face the mirror viewpoint. This ensures that the slice (SL) faces the user when viewed through the mirror. That is, the slice (SL) is oriented to face the user's viewpoint (V) derived by the viewpoint tracking system (VTS). The orientation of the slice (SL) is rendered using the aforementioned DMVF technique. That is, in order to accurately consider the user's viewpoint (V) relative to the display device 100, the slice (SL) is rotated relative to the plane (P) of the display device 100 and projected smaller or larger.

[0119] Examples of the VFSV technique from two different first-person viewpoints are shown in Figures 12A and 12B, with tools 110 and 20 remaining stationary in both Figures 12A and 12B. In Figure 12B, the user's viewpoint (V) moves slightly to the left, approaching tools 110 and 20. The distance from the user's viewpoint (V) to the display device 100 remains virtually unchanged. In this example, the slice (SL) is rotated to correspond to the horizontal movement of the user's viewpoint (V), so that the displayed slice (SL) appears the same to the user. If the user moves closer to the display device 100 in Figure 12B, the slice (SL) is displayed larger accordingly, and vice versa.

[0120] While the VFSV technique is being implemented using the user's viewpoint (V), the orientation of the computer image (R) of the slice (SL) may be further adjusted based on the pose of the display device 100, the pose of the object (O), and / or the pose of the tracked tools 110, 20 brought into the viewing frustum (F) of the virtual camera (VC) (i.e., the pose of the display device 100, the pose of the object (O), or the pose of the tracked tools 110, 20 brought into the viewing frustum (F) of the virtual camera (VC), or all of them).

[0121] Additionally, the control system 60 is configured to render the computer image (R) in accurate context for the VFSV technique. In other words, the computer image (R) rendering takes into account the relative spatial layering of objects (O), tools 110, 20, and / or slices (SL) (i.e., objects (O), tools 110, 20, or slices (SL), or all of them) in a common coordinate system, thereby simulating their actual spatial positions in the real coordinate system in which these items exist. Thus, any of the above descriptions regarding context-accurate rendering for the SPSV technique, including the visualizations in Figures 6 and 7, can be applied in their entirety to the VFSV technique and will not be repeated for the sake of brevity.

[0122] [C. Switching Visualization Modes] The control system 60 can be switched at any time between any of the visualization modes described herein, depending on any input or condition. For example, the control system 60 may be switchable between FVF visualization and DMVF visualization, SPSV visualization and VFSV visualization, 3D model visualization using FVF visualization (without slicing) and visualization using the SPSV technique (with or without 3D model), 3D model visualization using the DMVF technique (without slicing) and visualization using the VFSV technique (with or without 3D model), static visualization and any of the FVF technique, SPSV technique, DMVF technique, and VFSV technique.

[0123] The visualization mode may be switched in response to user input. For example, the control system 60 may receive a command from any of the input devices 106 described herein, which include, but are not limited to, a foot pedal 106a, a handheld pendant 106b, a display input 106c such as a tablet, smartphone, display device 100 or navigation system display, tool 110, 20 and / or head-mounted device 120 (i.e., a tablet, smartphone, display device 100 or navigation system display, tool 110, 20, or head-mounted device 120, or all of them), a push button, a sensor, a switch, a keyboard, a mouse, a microphone (for voice control), a gesture control device, a touchscreen, a joystick, and the like.

[0124] Switching between visualization modes may be performed automatically depending on conditions such as, but are not limited to, the type of surgical procedure, a specific step of the procedure, the user's viewpoint (V) or the tracked pose of the head-mounted device 120, the tracked pose of the object (O), the tracked pose of the display device 100, the tracked poses of the tools 110, 20, or any combination thereof.

[0125] Several embodiments have been discussed in the preceding description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive. In light of the above teachings, many variations and modifications are possible, and the invention may be carried out in ways other than those specifically described.

[0126] [D. Case studies and experimental results] To prove the concept, a secret case study was conducted with three trauma surgeons (chief surgeon, attending physician, and resident). The setup included a 3D-printed patient phantom, a display device 100 on an adjustable desk mount fixed to an OR table, the patient phantom, tools 110 and 20, the display device 100, and a Stryker® Flashpoint 6000 tracking system for tracking viewpoints. The 3D-printed patient phantom contained segmented anatomical structures and corresponding CT volumes. During the experiment, the surgeons investigated the proposed techniques in a predetermined order, without a think-aloud protocol or time constraints, followed by a semi-structured interview. The surgeons were asked to pay attention to the differences in visualization methods and consider possible application scenarios. First, the surgeons were instructed to use conventional orthogonal slice visualization controlled by the tracked instrument. Next, the surgeons were presented with the novel FVF visualization technique and SPSV visualization technique. Subsequently, surgeons investigated methods using head tracking, namely DMVF visualization and VFSV visualization techniques. When using FVF and DMVF, surgeons were instructed to switch between two modes: segmented 3D structures and direct volume rendering. In interviews, surgeons praised the idea of ​​automating slice presentation and the idea of ​​presenting surgeons with images that match the orientation of a direct view to the patient. Overall expert feedback on the proposed concepts was very positive. Surgeons appreciated the fact that DMVF and VFSV allow surgeons to interact with patient data in an aseptic manner, which they considered important if the intervention proved to be more complex than initially anticipated. Participants agreed that the FVF and SPSV methods could be easily integrated into established surgical workflows. When asked about possible applications of the proposed concepts, responses included use cases in complex craniofacial surgery, orthopedic and trauma surgery interventions, and interventions requiring high precision, such as internal prostheses.Finally, participants agreed that the proposed visualization technique could be helpful in familiarizing themselves with the anatomical structure of a particular patient and understanding how the instrument is currently positioned in relation to the planned trajectory. Participants unanimously agreed that the new visualization technique would be helpful in orientation regarding an overview of the anatomical structure of a particular patient and understanding how the instrument is currently positioned. The claims as originally filed are as follows: Claim 1: A display device is positioned on the first side of the physical object, defining a plane. Navigation system connected to control system and A system for supporting interaction with physical objects that have the following features: The aforementioned navigation system Aligning a computer image with the aforementioned physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device, It is configured to perform, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in accordance with the adjustment of the pose of the display device. A system to support interaction with physical objects. Claim 2: The system according to claim 1, wherein the computer image of the physical object is derived from a 3D model, and the control system is configured to control the display device to display one or more slices of the 3D model in accordance with the tracked pose of the physical object and the display device. Claim 3: The system according to claim 2, wherein the one or more slices are sliced ​​in a plane parallel to the plane of the display device. Claim 4: The system according to claim 2 or 3, wherein the control system is configured to control the display device in response to one or more of the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical instrument, and the tracked pose of the user's viewpoint, thereby automatically changing one or more slices to other slices. Claim 5: The system according to any one of claims 1 to 4, wherein the virtual position of the virtual camera is positioned at a predetermined distance from the plane of the display device, and the predetermined distance is automatically updated. Claim 6: The system according to claim 5, wherein the predetermined distance is automatically updated in accordance with at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, and a specific step of the surgical procedure. Claim 7: The system according to any one of claims 1 to 4, wherein the virtual position of the virtual camera is located at a predetermined distance from the plane of the display device, and the predetermined distance is fixed. Claim 8: The system according to any one of claims 1 to 7, wherein the XY arrangement of the virtual position of the virtual camera with respect to the plane of the display device is automatically updated. Claim 9: The system according to claim 8, wherein the XY configuration is automatically updated in accordance with at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, and a specific step of the surgical procedure. Claim 10: The system according to any one of claims 1 to 7, wherein the XY arrangement of the virtual position of the virtual camera with respect to the plane of the display device is fixed. Claim 11: The system according to any one of claims 1 to 10, wherein the pose of the display device is manually adjustable, and the virtual position of the virtual camera is automatically updated in accordance with the manual adjustment of the pose of the display device. Claim 12: The system according to any one of claims 1 to 11, comprising one or more actuators connected to the display device, and the control system configured to control the one or more actuators to adjust the pose of the display device. Claim 13: The system according to claim 12, further comprising an input device connected to the control system, wherein the control system is configured to receive commands from the input device and, in accordance with the commands, control one or more actuators to adjust the pose of the display device. Claim 14: The system according to claim 12 or 13, comprising a surgical instrument having one or more trackable features, wherein the navigation system is configured to track the pose of the surgical instrument in the common coordinate system and to control the display device to display an image of the surgical instrument, and the control system is configured to control the one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical instrument. Claim 15: The system according to any one of claims 12 to 14, comprising a viewpoint tracking system connected to the navigation system and configured to track the pose of the user's viewpoint in the common coordinate system, wherein the control system is configured to control one or more actuators to adjust the pose of the display device based on the tracked pose of the user's viewpoint. Claim 16: The system according to any one of claims 1 to 15, wherein the line defined between the virtual position of the virtual camera and the plane of the display device crosses the plane of the display device. Claim 17: The system according to any one of claims 1 to 16, wherein the line defined between the virtual position of the virtual camera and the plane of the display device is perpendicular to the plane of the display device. Claim 18: The system according to claim 16 or 17, wherein the line is at the geometric center of the display device. Claim 19: The system according to claim 16 or 17, wherein the line is located at a position offset from the geometric center of the display device. Claim 20: A system for assisting interaction with a physical object, comprising a display device positioned on a first side of the physical object and a navigation system connected to a control system, wherein the system defines a plane and includes these components. Aligning a computer image with the aforementioned physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device. Includes, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and automatically updating the virtual position of the virtual camera in accordance with the adjustment of the pose of the display device. How to activate a system to support interaction with physical objects. Claim 21: It is usable in a system that defines a plane and comprises a display device positioned on the first side of a physical object, and a navigation system connected to a control system, When executed by one or more processors, Aligning a computer image with the aforementioned physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device. It includes multiple instructions configured to execute, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the plane of the display device, and the virtual position of the virtual camera is automatically updated in accordance with the adjustment of the pose of the display device. A computer program product designed to assist with interaction with physical objects. Claim 22: A display device that defines a plane and in front of the plane where a physical object is positioned, Navigation system connected to control system and A system for supporting interaction with physical objects that have the following features: The aforementioned navigation system Aligning a computer image with the aforementioned physical object, Tracking the aforementioned physical object, the aforementioned display device, and the user's viewpoint pose in a common coordinate system, Controlling the display device to render the aligned computer image according to the physical object, the display device, and the tracked pose of the user's viewpoint. It is configured to perform, The rendering perspective is based on the field of view of a virtual camera whose virtual position is behind the plane, and the virtual position of the virtual camera is automatically updated in accordance with the movement of the tracked pose of the user's viewpoint. A system to support interaction with physical objects. Claim 23: The system according to claim 22, wherein the perspective of the rendering is automatically rotated to align with the tracked pose of the user's viewpoint relative to the plane of the display device. Claim 24: The rotation of the rendering is

number

number

Claims

1. A display device is positioned on the first side of a physical object, defining one front side. Navigation system connected to control system and A system for supporting interaction with the physical object comprising, The aforementioned navigation system Aligning the computer image of the physical object with the physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device. It is configured to perform, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the front of the display device, and the virtual position of the virtual camera is automatically updated in accordance with the adjustment of the pose of the display device. A system to support interaction with physical objects.

2. The system according to claim 1, wherein the computer image of the physical object is derived from a 3D model, and the control system is configured to control the display device to display one or more slices of the 3D model in accordance with the physical object and the tracked pose of the display device.

3. The system according to claim 2, wherein the one or more slices are sliced ​​in a plane parallel to the front surface of the display device.

4. The system according to claim 2 or 3, wherein the control system is configured to control the display device in response to one or more of the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, and the tracked pose of the user's viewpoint, thereby automatically changing one or more slices to other slices.

5. The system according to claim 1, wherein the virtual position of the virtual camera is located at a predetermined distance from the front of the display device, and the predetermined distance is automatically updated.

6. The system according to claim 5, wherein the predetermined distance is automatically updated in accordance with at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, and a specific step of the surgical procedure.

7. The system according to claim 1, wherein the virtual position of the virtual camera is located at a predetermined distance from the front of the display device, and the predetermined distance is fixed.

8. The system according to claim 1, wherein the XY arrangement of the virtual position of the virtual camera relative to the front of the display device is automatically updated.

9. The system according to claim 8, wherein the XY configuration is automatically updated in accordance with at least one of the following: the tracked pose of the display device, the tracked pose of the physical object, the tracked pose of the surgical tool, the tracked pose of the user's viewpoint, the type of surgical procedure, and specific steps of the surgical procedure.

10. The system according to any one of claims 1 to 7, wherein the XY arrangement of the virtual position of the virtual camera relative to the front of the display device is fixed.

11. The system according to any one of claims 1 to 10, wherein the pose of the display device is manually adjustable, and the virtual position of the virtual camera is automatically updated in accordance with the manual adjustment of the pose of the display device.

12. The system according to any one of claims 1 to 11, comprising one or more actuators connected to the display device, and the control system configured to control the one or more actuators to adjust the pose of the display device.

13. The system according to claim 12, further comprising an input device connected to the control system, wherein the control system is configured to receive commands from the input device and, in accordance with the commands, control one or more actuators to adjust the pose of the display device.

14. The system according to claim 12 or 13, comprising a surgical tool having one or more trackable features, wherein the navigation system is configured to track the pose of the surgical tool in the common coordinate system and to control the display device to display an image of the surgical tool, and the control system is configured to control one or more actuators to adjust the pose of the display device based on the tracked pose of the surgical tool.

15. The system according to any one of claims 12 to 14, comprising a viewpoint tracking system connected to the navigation system and configured to track the pose of the user's viewpoint in the common coordinate system, wherein the control system is configured to control one or more actuators to adjust the pose of the display device based on the tracked pose of the user's viewpoint.

16. The system according to any one of claims 1 to 15, wherein the virtual position of the virtual camera is defined by a virtual line between the virtual position and the front surface of the display device, and the line traverses the front surface of the display device.

17. The system according to any one of claims 1 to 15, wherein the virtual position of the virtual camera is defined by a virtual line between the virtual position and the front surface of the display device, and is perpendicular to the front surface of the display device.

18. The system according to claim 16 or 17, wherein the line is at the geometric center of the display device.

19. The system according to claim 16 or 17, wherein the line is located at a position offset from the geometric center of the display device.

20. A system for assisting interaction with a physical object, comprising a display device that defines one front surface and is positioned on the first side of the physical object, and a navigation system connected to a control system, Aligning a computer image with the aforementioned physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device. Includes, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the front of the display device, and automatically updating the virtual position of the virtual camera in accordance with the adjustment of the pose of the display device. How to activate a system to support interaction with physical objects.

21. It is usable in a system comprising a display device that defines one front and is positioned on the first side of a physical object, and a navigation system connected to a control system, When executed by one or more processors, Aligning a computer image with the aforementioned physical object, Tracking the poses of the physical object and the display device in a common coordinate system, Controlling the display device to render the aligned computer image according to the tracked pose of the physical object and the display device. Includes multiple instructions configured to execute, The rendering perspective is based on a virtual camera whose virtual position is located on a second side of the physical object opposite to the first side, the virtual camera having a field of view facing the front of the display device, and the virtual position of the virtual camera is automatically updated in accordance with the adjustment of the pose of the display device. A computer program product designed to assist with interaction with physical objects.

Images