Bidirectional communication between the head-mounted display and the electroanatomical system

Abstract

The processing system provides bidirectional communication between an electroanatomical navigation and mapping system (EAMS) and a head mounted display (HMD). A user wearing an HMD can use hands-free or touch-free actions to interact with a user interface of EAMS information displayed by the HMD (e.g., 3D augmented reality graphics) or by a display monitor (e.g., 2D graphics). For example, the processing system updates the position of a cursor on the display monitor to track the user's gaze direction as determined by the HMD.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 333,900, filed April 22, 2022, which is incorporated by reference in its entirety for all purposes.

[0002] The present disclosure relates generally to a system for electroanatomical mapping and display manipulation in a virtual reality, augmented reality, or mixed reality environment. [Background technology]

[0003] Electroanatomic navigation and mapping systems (EAMS) aid physicians performing minimally invasive medical procedures. EAMS tracks the distal end of a medical instrument, such as a catheter. EAMS displays the position of the medical instrument in relation to the patient's anatomy and other patient data via real-time computer-generated images displayed on a (monoscopic) 2D computer screen (e.g., an LCD flat panel mounted on a boom above the patient).

[0004] Traditionally, a user operates an EAMS using a keyboard and a mouse. Typically, the hands of a doctor performing a medical procedure are sterile and occupied with the task of holding and manipulating medical instruments. Therefore, the doctor cannot directly interact with the EAMS. Instead, another person (e.g., a technician) operates the EAMS for the doctor. The doctor communicates requests to the EAMS technician via voice. Summary of the Invention

[0005] There is an advantage to supplementing the EAMS with or connecting the EAMS to a head-mounted display (HMD) worn by the user that tracks the position and orientation of the user's head and / or eyes / pupils, such as the Sentiar™ SentEP. The EAMS sends data about the patient's anatomy as well as about the position and status of the catheter or other medical instrument being tracked. The HMD generates a 3D image of this data from the EAMS and displays the 3D image to the physician. The user can rotate or otherwise change their view of the EAMS data by simply moving their head, instead of having to verbally ask the EAMS technician to manipulate the view with a mouse or keyboard. Additionally, the HMD can also present a user interface (e.g., menus and buttons) that can be controlled using a cursor tied to where the user is pointing with their head or eyes / pupils. This allows for further interaction with the received EAMS data, such as zooming in and out, and turning on or off various visual elements. Benefits of the embodiments described herein include, among other advantages, providing the user with better understanding of complex three-dimensional spatial relationships in EAMS data, greater accuracy in positioning a catheter or other medical instrument, and reduced need for verbal communication with the EAMS technician.

[0006] The HMD provides the aforementioned benefits by receiving data from the EAMS as well as allowing the user to view the EAMS data and manipulate the view of the data. In conventional systems, the HMD user cannot affect the state or data of the EAMS itself. If the user wants to change the data on the EAMS itself (e.g., edit the shape of the patient's anatomy), the user must verbally request that the technician perform the operation on the user's behalf. Verbally communicating complex spatial data using commands is difficult and cumbersome, and there is a risk that the technician will misunderstand some nuance in the user's communication.

[0007] The embodiments described herein describe novel systems and methods that utilize bidirectional communication between an HMD and an EAMS and the benefits of doing so compared to conventional systems that use, for example, unidirectional communication from the EAMS to the HMD.

[0008] In various embodiments, the system includes a head mounted display (HMD) worn by a user, a display monitor, and a processing system configured to provide image data for display by the HMD based on data provided to the display monitor for a user interface displayed by the display monitor, receive a user's gaze direction from the HMD, determine a position of a cursor displayed in the user interface based on the gaze direction of the HMD user, provide the cursor position to the display monitor to update the display of the cursor on the display monitor, receive user input from the HMD in response to actions performed by the user, and provide user input information to the display monitor to update the user interface based on the user input and the cursor position.

[0009] In some embodiments, the processing system includes a frame grabber configured to receive data to be provided to the display monitor for a user interface to be displayed by the display monitor.

[0010] In some embodiments, the processing system includes a first computing device, a second computing device configured to provide image data for display by the HMD and receive user input from the HMD, and a signal generator configured to transmit information related to the user input to the first computing device, where the first computing device generates user input information using the information related to the user input.

[0011] In some embodiments, the signal generator transmits information related to the user input as a USB signal, and the first computing device generates the user input information without using input from a mouse or keyboard.

[0012] In some embodiments, the second computing device receives the user's gaze direction from the HMD and the first computing device provides the position of the cursor to the display monitor.

[0013] In some embodiments, the HMD is further configured to display the augmented reality graphics and, in response to determining that a user is interacting with the user interface, update the display of the augmented reality graphics to track a cursor position.

[0014] In some embodiments, the processing system is further configured to determine that the action performed by the user is a selection of one of a plurality of user controls displayed in the user interface.

[0015] In some embodiments, the processing system is further configured to provide the registration information to the 3D anatomical information source.

[0016] In various embodiments, the method includes providing image data for display by a head mounted display (HMD) worn by a user based on data provided to the display monitor for a user interface displayed by the display monitor, receiving a user's gaze direction from the HMD, determining a position of a cursor displayed in the user interface based on the HMD user's gaze direction, providing the cursor position to the display monitor to update a display of the cursor on the display monitor, receiving user input from the HMD in response to an action performed by the user, and providing user input information to the display monitor to update the user interface based on the user input and the cursor position.

[0017] In some embodiments, the method further includes receiving data to be provided to the display monitor for a user interface to be displayed by the display monitor.

[0018] In some embodiments, the method further includes transmitting information related to the user input from the second computing device to the first computing device by a signal generator, wherein the first computing device generates user input information using the information related to the user input.

[0019] In some embodiments, the signal generator transmits information related to the user input as a USB signal, and the first computing device generates the user input information without using input from a mouse or keyboard.

[0020] In some embodiments, the second computing device receives the user's gaze direction from the HMD and the first computing device provides the position of the cursor to the display monitor.

[0021] In some embodiments, the method further includes displaying, by the HMD, the augmented reality graphic, and in response to determining that a user is interacting with the user interface, updating the display of the augmented reality graphic to track a cursor position.

[0022] In some embodiments, the method further includes determining that the action performed by the user is a selection of one of a plurality of user controls displayed in the user interface.

[0023] In some embodiments, the method further comprises providing the registration information to a 3D anatomical information source.

[0024] In various embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform any step of the methods described herein.

[0025] In some embodiments, the processing system including the HMD sends live audio to the EAMS for internal communication with remote users, dictation of notes and procedure annotations, or voice control of EAMS functionality.

[0026] In some embodiments, the processing system, including the HMD, can remotely operate functionality otherwise provided to the EAMS technician via a keyboard and mouse. Exemplary functionality includes having the EAMS start and stop collecting catheter-tissue contact points to generate or update the geometry of the patient's anatomy, switching which catheter / instrument is being used for mapping, and displaying ultrasound or fluoroscopy images. designating specific frames of an ultrasound or fluoroscopic image to be saved for later review; annotating features in the saved ultrasound or fluoroscopic images; changing the level of detail at which the EAMS generates the patient's anatomy geometry; designating small regions of the anatomy to the EAMS to be selected for operations such as cutting, trimming, deleting, smoothing, pushing inward or outward; instructing the EAMS to perform operations on selected small regions of the patient's anatomy, such as cutting, trimming, deleting, smoothing, pushing inward or outward; it designates to the EAMS how the current anatomical geometry or electrical map should be integrated with others previously acquired, including those acquired from ultrasound, CT, and MRI images; ), specify which electrodes on the catheter to highlight or turn on / off, specify which visualization types or styles it should display on its 2D screen and which it should send to the HMD headset, specify thresholds and end points for data to color mapping, command the EAMS to start and stop video visualization, specify the EAMS to create marker points in space and / or time and make notes or annotations on them for later editing or review (points can be relative to the patient's anatomy, catheter to instrument tip location, location in an ultrasound, CT, MRI, or X-ray image, or signal features in an electrogram), and specify which anatomical samples, markers,Instructing the EAMS to select, edit or delete tags or labels; instructing the EAMS to take a snapshot image or video of the current EAMS screen, data, for later playback review or export to the patient medical record; instructing the EAMS to measure distances between points and features in the anatomy (virtual calipers); instructing the EAMS to measure timing differences between timestamps and features in an electrogram waveform; instructing the EAMS to rewind, fast forward, pause and stop playback of previously recorded data; and instructing the EAMS to create annotations, e.g., lines and curves projected onto the 3D surface of the anatomy.

[0027] In some embodiments, the position and orientation of the user's head and / or gaze are sent by the HMD to the EAMS for the purposes of: displaying what the user is looking at (in the HMD) on the EAMS 2D display monitor, sharing what the user is looking at with other users wearing remote EAMS and HMDs, recording what the user is looking at in recording medical procedure data, or moving computational tasks and workload from the HMD to the EAMS computing device (e.g., remote rendering with time warp or late reprojection, or re-sorting voxels and / or polygons by distance from the user's viewing position).

[0028] In some embodiments, the battery and / or temperature status of the HMD is sent to and displayed by the EAMS to aid the user in preparing for swapping in another HMD battery and troubleshooting.

[0029] In some embodiments, the processing system measures the throughput, jitter, latency, signal to noise ratio, and / or loss rate of the wireless network between the EAMS and the HMD so that the EAMS can reduce or increase the amount of data sent to the EAMS to manage the delay between when the EAMS sends the data and when the HMD displays it. The processing system can implement mesh decimation, progressive refinement of 2D or 3D images, or foveated rendering, with the highest fidelity portion of the image being where the user is looking or where the catheter tip (or another part of the medical instrument) is moving. [Brief description of the drawings]

[0030] The disclosed embodiments have advantages and features that will become more readily apparent from the detailed description, the appended claims, and the accompanying figures (or drawings), a brief introduction of which is given below.

[0031] [Figure 1] FIG. 1 illustrates an exemplary system environment for a processing system according to one embodiment. [Diagram 2] 1 is a flow diagram of a process for user control in an augmented reality environment according to one embodiment. [Diagram 3] FIG. 1 illustrates an exemplary user interface in an augmented reality environment according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] As disclosed herein, embodiments of two-way EAMS communication provide a user wearing an HMD with greater and improved control of the user interface or other functionality.

[0033] In some embodiments, the HMD provides a user interface (e.g., buttons and menus) that can be controlled hands-free by a user, such as a physician, and that can enable the physician to operate functions of the EAMS that would otherwise require the physician to make verbal requests to an EAMS technician.

[0034] Exemplary commands that the HMD can send to the EAMS include starting or stopping collection of contact points between the medical instrument and tissue to generate or update information of the patient's anatomy (e.g., geometry or shape), designating specific frames of ultrasound or fluoroscopic images to be saved for later review, annotating features within saved ultrasound or fluoroscopic images, switching which medical instrument is used for mapping, turning on or off collection of electrograms (EGMs), performing actions on selected subregions of the patient's anatomy, such as cutting, trimming, deleting, smoothing, pushing inward or outward, and displaying animated visualizations (e.g., BWI). These functions include starting or stopping the display of EAMS (e.g., CARTO ripple maps), taking snapshot images or video, data of the current EAMS screen for later playback review or export to the patient medical record, measuring distances between points and features in the anatomy (e.g., virtual calipers), measuring timing differences between timestamps and features in an electrogram waveform, and requesting to rewind, fast forward, pause, and stop playback of previously recorded data.

[0035] The HMD can send information to the EAMS to update the display on the HMD. Example information includes the level of detail at which the EAMS generates the patient's anatomical geometry (e.g., changing to a more simplified or detailed level); small regions of the anatomy to be selected for actions such as cutting, trimming, deleting, smoothing, pushing inward or outward; instructions for integrating the anatomical geometry or electrical map with previously acquired data, e.g., acquired from an ultrasound, CT, or MRI image; which electrodes on the catheter to highlight or turn on / off; what type or style of visualization (e.g., map) should be displayed on a 2D display monitor or HMD (e.g., in 3D); thresholds and end points of the data-to-color mapping; marker points in space or time and the creation of associated notes or annotations for later editing or review (marker points can be relative to the patient's anatomy); instrument tip position from the catheter; medical instrument position within the ultrasound, CT, MRI, or X-ray image; signal features within the electrogram; and which anatomical samples, markers, tags, or labels to select, edit, or delete.

[0036] In various embodiments, the functionality performed by the EAMS technician using a mouse and keyboard can also be controlled by the user wearing the HMD.

[0037] I. Electroanatomical Navigation and Mapping System (EAMS) In an electrophysiology procedure, an operator user and / or electrophysiologist user may use a 2D display monitor, mouse, and keyboard to specify a three-dimensional spatial relationship with three or six degrees of freedom (e.g., 3D position, direction and magnitude, or 3D position and 3D orientation in 3D space) with any number of supporting computer systems providing patient data. An example implementation is an EAMS. Any number of supporting systems may provide additional procedure data or equipment control adjacent to or within the EAMS data.

[0038] In some procedures, the user imports a 3D model of the patient's anatomy (derived using EAMS from a pre-operative CT, MRI, or ultrasound scan, or from a previously performed cardiac mapping procedure). The user then specifies how the imported 3D model spatially relates to the current intra-operative cardiac 3D model. Typically, the user must identify three or more corresponding 3D points on the surface of the model. The EAMS then calculates the most appropriate transformation between the two models. In other situations, the user uses the mouse to interactively apply any number of translation, rotation, and / or scaling operations to the model using a 2D display monitor, mouse, and / or keyboard.

[0039] In some procedures, the catheter is robotically manipulated in part or in whole via any number of mechanical cables, motors, or magnetic or pneumatic actuators, either internal or external to the catheter. The user specifies a 3D trajectory vector, a 3D course, and / or a 3D target destination location. The robotic system then calculates the appropriate actuations to advance and move the catheter as specified by the user.

[0040] In some ablation systems, the direction in which the ablation energy is directed can be steered electronically or mechanically (e.g., high intensity focused ultrasound, phased array RF antennas). The user specifies a 3D vector along which the ablation energy is directed from an emission point on the catheter.

[0041] In the previous example, the EAMS user is using a 2D display monitor and a mouse with two degrees of movement, so the user can only observe and adjust two degrees of freedom of the desired 3D spatial relationship at a time (there can be three to six degrees of freedom). The user frequently cycles between modes: (1) adjusting translation in 2D, (2) adjusting rotation in 2D, and (3) changing the looking direction (i.e., pitch and yaw) in 2D. The user will need to frequently cycle between these three modes to interpret, build, adjust, and inspect the ongoing 3D spatial relationship until it sufficiently resembles the user's intent. This operation requires practice and skill, and can be tedious, time-consuming, and error-prone.

[0042] II. System Overview 1 illustrates an exemplary system environment for a processing system 100 according to one embodiment. The system environment includes a processing system 100, one or more HMDs 110, one or more display monitors 120, and one or more input devices 130. The processing system 100 includes an EAMS computing device 102 (also referred to herein as "EAMS"), a data computing device 104, a frame grabber 106, and a signal generator 108. In other embodiments, the functionality of the processing system 100 may be implemented by any number of devices. For example, instead of an EAMS computing device 102 separate from the data computing device 104, the processing system 100 includes a single computing device (including memory and one or more processors) that implements the functionality of both the EAMS computing device 102 and the data computing device 104, as well as the functionality of the frame grabber 106 and the signal generator 108.

[0043] In some embodiments, the functionality of the HMD 110 and the EAMS computing device 102 is implemented on separate devices. One computing device is part of the EAMS and another computing device is part of the HMD 110. In other embodiments, the functionality of the two computing devices is combined into a single computing device. In other embodiments, some of the functionality of the EAMS computing device 102 or the HMD 110 is implemented on a remotely located server, which may be part of the processing system 100.

[0044] The EAMS computing device 102 is communicatively coupled to a data computing device 104, for example, via the Internet or another form of wireless or wired connection. The EAMS computing device 102 is communicatively coupled to one or more input devices 130, such as a keyboard, mouse, or 3D anatomical information source. The EAMS computing device 102 transmits information for display on a display monitor 120 (e.g., a computer monitor having a display such as an LCD, LED, OLED, plasma, or touch screen display). A frame grabber 106 receives the information transmitted to the display monitor 120 and transmits it to the data computing device 104.

[0045] The data computing device 104 is communicatively coupled to one or more HMDs 110 via a wireless or wired connection. In a conventional system, the EAMS computing device 102 receives input from an input device such as a mouse or keyboard. The signal generator 108 can provide input to the EAMS computing device 102 that resembles the input from a mouse or keyboard. The signal generator 108 uses information from the data computing device 104 to generate an input signal (e.g., a USB signal). An example use case is described below with respect to Figures 2 and 3.

[0046] Figure 2 is a flow diagram of a process 200 of user control in an augmented reality environment according to one embodiment. Figure 3 shows an exemplary user interface in an augmented reality environment according to one embodiment. The processing system 100 performs the process 200, which may be used for a cardiac medical procedure, as shown in the exemplary user interface shown in Figure 3. In other embodiments, the process 200 is also applied to other types of medical procedures.

[0047] In step 210, the processing system 100 provides image data for display by the HMD 110 based on data provided to the display monitor 120 for the user interface 120 to be displayed by the display monitor 120. In some embodiments, the data computing device 104 may perform step 210, and the EAMS computing device 102 provides data to the display monitor 120 of the user interface. The data computing device 104 receives the data provided to the display monitor 120 via the frame grabber 106. As shown in FIG. 3, the HMD 110 displays an augmented reality graphic 300 that is a model of a patient's heart. The HMD 110 also displays other augmented reality graphics including a catheter 320 and user controls 330. As the user looks towards the display monitor 310, the HMD 110 displays the augmented reality graphic 300 and other graphics as holograms superimposed on the display monitor 310 from the user's perspective.

[0048] In step 220, the processing system 100 receives the user's gaze direction from the HMD 110. The HMD 110 determines the gaze direction using sensor data captured by one or more sensors of the HMD 110. For example, the sensor data may represent a head direction or an eye / pupil direction.

[0049] In step 230, the processing system 100 determines a position of a cursor displayed on the user interface based on the gaze direction of the HMD user. As shown in FIG. 3, the display monitor 310 displays a cursor 350 (e.g., a mouse cursor) on the user interface. Additionally, the HMD 110 displays a marker 340 to indicate the gaze direction of the HMD user. The processing system 100 can control the position of the cursor 350 based on the position of the marker 340. Specifically, in step 240, the processing system 100 provides the position of the cursor 350 to the display monitor 310 to update the display of the cursor 350 on the display monitor 310. In response to determining that the marker 340 is located within an area of ​​the user interface displayed by the display monitor 310, the processing system 100 updates the position of the cursor 350 to track the position of the marker 340.

[0050] At step 250, the processing system 100 receives user input from the HMD 110 in response to an action performed by the user. The user input may be a hands-free input such as an eye gesture (e.g., a wink or blink), a head gesture (e.g., a nod), or a voice command. The processing system 100 may determine that the user input is an interaction with one or more user controls displayed in the user interface (e.g., a button, a slider, a menu, a text entry box).

[0051] In step 260, the processing system 100 provides user input information to the display monitor 310 to update the user interface based on the user input and the position of the cursor 350. As an example, the user performs a user input to start or stop a mapping or registration procedure. Instead of using a mouse or keyboard to interact with the user interface on the display monitor, the user can use hands-free input to start or stop the procedure by directing the user's gaze to a virtual button (e.g., an augmented reality graphic) displayed by the HMD 110 or to a button on the display monitor (where the cursor tracks the user's gaze direction as indicated by the marker 340 shown in FIG. 3). Instead of using a mouse click or pressing a keyboard key, the user provides hands-free input such as a wink or blink to indicate the selection of a virtual button or buttons on the display monitor. The signal generator 108 transmits information related to the user input from the HMD 110 to the EAMS computing device 102, which processes the information when it is to process the input from the mouse or keyboard received via the USB signal.

[0052] As another use case, a user performs a user input to store the current 3D location of the catheter tip as a marker or sample point. The data computing device 104 continuously calculates the 2D coordinates - in the coordinate frame of the user interface on the display monitor 120 - and the HMD gaze markers (e.g., markers 340 shown in FIG. 3) overlay the user interface from the viewpoint of the user wearing the HMD. The data computing device 104 then generates a USB mouse signal to update the cursor on the display monitor 120 to those coordinates so that the cursor appears to follow the HMD gaze markers. The user moves the cursor to a pull-down menu in the user interface by winking or performing another type of touchless action and selects the pull-down menu. The user can again use the gaze direction and another action to select an option from the menu, such as "Store sample point."

[0053] In various embodiments, the EAMS and HMD can send data via wired or wireless transmission, including USB, Ethernet, PCI, Bluetooth, Wi-Fi, ultra-wideband RF, optical links, and using topologies such as direct point-to-point, token ring, mesh, or via a central router. Examples of HMD devices include optical see-through augmented reality (AR) headsets (e.g., Microsoft HoloLens, ODGR-7, Magic Leap 1 and 2, or Nreal Light), camera-based see-through AR headsets (e.g., Oculus Quest, Varjo XR-3, or Lynx R1), fully opaque virtual reality (VR) headsets (e.g., HTC Vive and Valve Index), and other displays that can track the user's head or gaze direction and the display's position or orientation from the user's perspective.

[0054] Although the embodiments described herein refer to a head mounted display (HMD) 110, the disclosed systems and methods may also be applied to non-head mounted displays (e.g., fixed to a floor, ceiling, wall, table, or cart) that are monoscopic or stereoscopic (e.g., Looking Glass 8k Gen 2, Sony LMD-X310MT, or Steris VividImage 4k Surgical Display) and coupled with a head or eye tracker (e.g., Tobii Pro Fusion, Tobii Pro Spectrum, or Tobii Tracker 5). In some embodiments, the system combines a head worn eye tracker (e.g., Tobii Pro Glasses 3, Pupil Labs Core, Argus Science, or ETVision) with a non-head mounted display.

[0055] In various embodiments, the user interface that captures user commands for the EAMS can use any combination of sensing modalities, including head position and head orientation, eye or pupil gaze direction, audio and voice input, hand or finger tracking, EEG (electroencephalography) or EMG (electromyography) sensors, facial gesture tracking (e.g., via cameras, depth sensors, ultra-wideband RF, or short-range radar), tongue switches, handheld remote controls, laser pointers, foot pedals, and buttons or joysticks attached to the display or proximal end of the catheter or medical instrument.

[0056] Examples of EAMS systems include Johnson and Johnson Biosense Webster CARTO™, St. Jude Medical Ensite™ Velocity™, Medtronic LocaLisa™, Medtronic CardioInsight, and Medtronic Affera.

[0057] In various embodiments, the ultrasound images processed by the EAMS or HMD are acquired by intracardiac, transesophageal, external, 2D or 3D ultrasound transducers with or without the use of ultrasound contrast agents.

[0058] In various embodiments, the EAMS includes one or more navigation systems, such as those used for procedures involving one or more of the heart, brain, sinuses, jaw, teeth, spine, knee, hip, lung, prostate, liver, kidney, blood vessels, prostate, pancreas, uterus, and abdomen.

[0059] The embodiments described herein may be used in conjunction with any number of catheters or other medical instruments that are manually operated by a user's hand or assisted or controlled by a motor or other mechanical actuator (e.g., a robotically controlled or power-assisted medical instrument).

[0060] The disclosure herein generally refers to users as physicians, but the disclosed embodiments are applicable to other types of users, including veterinarians, nurses, trainees, instructors, technicians, engineers, administrators, and demonstrators, among other types of health care providers.

[0061] III. Head-Mounted Display (HMD) In various embodiments, the HMD 110 comprises one or more displays and user input devices. In some embodiments, user input is achieved through one or more microphones. In some embodiments, user input is achieved through hand position and posture observation (e.g., using a camera or neuromuscular sensing). In some embodiments, user input is achieved through the relative position and posture of the HMD (e.g., gaze or head position cursor). In some embodiments, user input is achieved through sensors communicatively coupled to provide any number of button inputs, hand position, posture, or neuromuscular cues. In some embodiments, user input is achieved by tracking eye position and posture (e.g., a camera, neuromuscular sensing).

[0062] When viewed in a stereoscopic head-tracked mixed reality HMD, the ongoing 3D spatial relationship can be much more salient to the user than when shown on a 2D display. Furthermore, the ongoing 3D spatial relationship can be simultaneously adjusted as it is inspected, without the user having to switch modes.

[0063] IV.3D Registration In one embodiment, a system comprising an HMD 110 worn by a user, a 2D display monitor 120, and a processing system 100 configured to provide any number of 3D anatomical information and responsive to transformations between 3D anatomical coordinate systems allows a user to intuitively specify, update, and / or observe the registration of the 3D anatomical information in the combined coordinate system.

[0064] In some embodiments, 3D anatomical information (e.g., registration information) is generated and transmitted intraoperatively by the EAMS and transmitted to processing system 100 for registration with another source of 3D anatomical information (e.g., input device 130). In other embodiments, the 3D anatomical information is transmitted via a Picture Archiving and Communication System (PACS) for registration. In other embodiments, the 3D anatomical information is generated by processing system 100 by combining 3D information from one source (e.g., catheter position) with another source (e.g., electrogram recording system, ultrasound).

[0065] In some embodiments, the processing system 100 sends registration information to the 3D anatomical information source via emulation of inputs originating from the 3D anatomical information source to aid in registration (e.g., a series of any number of keyboard or mouse actions). These actions will apply the normal registration actions of the existing 3D anatomical information source. In other embodiments, the registration information is communicated from the processing system 100 to the 3D anatomical information source by providing a set of corresponding points from the source information source to the destination information source. In some embodiments, the registration information is communicated from the processing system 100 to the 3D anatomical information source by providing functionality for transforming any input position and / or pose from one anatomical information source to the other anatomical information source. In some embodiments, the transformations are simply contained and displayed within the processing system 100, the information is transformed within the processing system 100, and communication between the anatomical information sources is communicated within each system's native coordinate system.

[0066] V. Catheter manipulation In one embodiment, a system comprising an HMD 110 worn by a user and a processing system 100 configured to receive at least one source of 3D anatomical information and to communicate with at least one catheter manipulator enables a user to intuitively specify, update, and / or observe the operation of any number of catheters.

[0067] In some embodiments, a user specifies a target 3D position for the catheter manipulator. In some embodiments, a user specifies a series of target 3D positions for the catheter manipulator to create a target trajectory. In some embodiments, a user specifies a 3D position and target pose of the HMD manipulator to specify both the position and orientation of the catheter at any number of positions. In some embodiments, the processing system 100 communicates with the manipulator via an EAMS. In some embodiments, the processing system 100 communicates with the catheter manipulator independent of the EAMS.

[0068] VI. Steerable Catheters In one embodiment, a system comprising an HMD 110 worn by a user and a processing system 100 configured to receive at least one 3D anatomical information source and communicate with at least one configurable catheter ablation system allows a user to intuitively specify, update, and / or observe 3D parameters of the catheter ablation system. In some embodiments, the user specifies and observes directional sensing parameters (e.g., electrode selection in array, ultrasound). In some embodiments, the user specifies ablation direction parameters (e.g., electrodes in array, phase, laser direction). In some embodiments, the processing system 100 communicates with the catheter ablation system via an EAMS. In some embodiments, the processing system 100 communicates with the catheter ablation system independent of the EAMS.

[0069] VII. User Input In some embodiments, user input from the HMD 110 is accomplished by detecting and transmitting any number of click actions to indicate discrete events (e.g., start, stop). In some embodiments, these click actions are accomplished hands-free and touch-free by holding the user head pose within a motion tolerance of the dwell period (gaze dwell). In other embodiments, these click actions are accomplished via a handheld controller in communication with the HMD 110 and / or processing system 100. In some embodiments, these actions are detected via processing of voice commands using a dictionary, or derived intents using any number of voice intent detection operations. In some embodiments, these actions are detected via physical gestures, including facial expressions, hand postures or movements, eye postures or movements, and the like. Physical gestures may be detected and classified as actions by observing neuromuscular signals. Additionally, physical gestures may be detected by processing sequences of image data from 2D sensors, including electro-optical, or infrared (IR). In some embodiments, these discrete inputs are divided into ranges of variable inputs to provide intermediate value ranges (e.g., [0-10]).

[0070] In various embodiments, 2D user input from the HMD 110 is accomplished by detecting and transmitting absolute or relative 2D position changes. In some embodiments, this 2D information is calculated by processing the position of the HMD gaze cursor relative to the extent of a virtual or physical display. In some embodiments, coordinates can be converted between the virtual and physical displays to facilitate manipulation by a user wearing the HMD 110, for example, electrograms can be multiplied in virtual scale to provide greater precision with hands-off control as per EGM notes.

[0071] In various embodiments, 3D user input from the HMD 110 is achieved by detecting and transmitting absolute or relative 3D position changes. In some embodiments, this 3D information is calculated by processing 3D position changes of a user wearing the HMD 110. In some embodiments, the 3D information is calculated by processing the position of a user's single hand input. In some embodiments, the 3D information is calculated through processing the difference of the input of the user's two hands. The hand input can be achieved through camera-based hand tracking or through handheld or hand-worn controller input. In various embodiments, 3D position and orientation information is available in addition to the 3D position. The processing system 100 can calculate this information from a single input, e.g., the position and orientation of one hand, or from a combination of inputs, e.g., the position of one hand and the relative position of another hand that form the basis of the orientation vector.

[0072] VIII. Multi-User Use Cases The above-described embodiments may also be used by multiple users, simultaneously or sequentially, to allow a collection of users to collaborate in specifying 3D spatial relationships. For example, one user wearing an HMD may make large, global adjustments to the ongoing 3D spatial relationships using a handheld controller, while another user wearing another HMD (who may have sterilized hands) may make smaller, precise refinements to the ongoing 3D spatial relationships. Similarly, an EAMS operator sitting at a desk may assist the user wearing an HMD by making large, global adjustments to the ongoing 3D spatial relationships using a mouse and keyboard, while a user wearing an HMD standing next to the patient may make smaller, more precise refinements to the ongoing 3D spatial relationships.

[0073] In some embodiments, when multiple users are given the ability to specify 3D spatial relationships, it may be easier for users to understand how to operate the processing system 100 if the system enforces that only a single user is allowed to virtually grasp a pre-operative model, a 3D trajectory, a 3D orientation and magnitude, or a 3D point at a time. Depending on whether the processing system 100 consists of a single or multiple processing units, a traditional shared memory mutex or an untrusted mutex algorithm (e.g., Suzuki-Kasami algorithm, Ricart-Agrawala algorithm) may be used to lock users out of virtually grasping a pre-operative model, a 3D trajectory, a 3D orientation and magnitude, or a 3D point until the current owner releases it.

[0074] In some embodiments, the HMD 110 must send the 3D spatial relationship back to the EAMS computing device 102. In some embodiments, the HMD 110 keeps the ongoing 3D spatial relationship private to itself until the user signals that they are finished specifying it, and then the HMD 110 sends the completed 3D spatial relationship to the EAMS computing device 102. In other embodiments, the HMD 110 continuously or periodically sends the ongoing 3D spatial relationship to the EAMS computing device 102 while the user is still manipulating and refining it.

[0075] IX. EAMS API In some embodiments, the EAMS API outputs the position, orientation and shape of the catheter, a geometric and visual description of the 3D heart model, and / or the waveforms of the signals sensed by the electrodes on the catheter. In addition to outputting these elements via the API, the EAMS typically displays these elements on its own 2D display. The HMD 110 can use data from the EAMS API to reconstruct the appearance of parts of the EAMS display monitor 120. This allows the HMD 110 to perform a pattern match between the reconstructed EAMS elements and a live video (with a view of the EAMS 2D display monitor) acquired by the camera of the HMD 110 instead of the desktop video captured by the frame grabber 106. This has the benefit of requiring fewer hardware components.

[0076] In some embodiments, the signal waveforms captured by the catheter electrodes may not only appear on the EAMS display monitor 120, but may also be digitized by a device external to the EAMS (e.g., GE Cardiolab). This external digitizer device (input device 130) can send the signal waveforms to the HMD 110 via the processing system 100 using a digitizer API that is independent of the EAMS API.

[0077] In some embodiments, the EAMS computing device 102 assists the HMD 110 in detecting EAMS video captured in the HMD 110 camera video by adding one or more additional 2D graphic elements to the EAMS screen shown by the display monitor 120 and sending a description of the additional 2D graphic elements via the EAMS API. The additional elements are designed to be more easily recognizable (by being high contrast and having unique characteristics) in the HMD camera video than in typical EAMS desktop video. Examples of additional 2D graphic elements include Aruco fiducials, QR codes, APRIL tags, and 2D barcodes.

[0078] Using two-way EAMS communication to improve collaboration with users viewing X.2D EAMS displays. Because conventional EAMS systems without an HMD use a 2D display, the EAMS technician will slightly rotate (rock) the view back and forth around the point of interest to give the viewer a better understanding of the displayed 3D spatial relationships. As an example, this is commonly done when a physician is using a catheter tip to apply radio frequency (RF) energy to a specific point in the patient's tissue and must carefully control the location of the tip within the anatomy. HMDs are stereoscopic, and therefore there is little need for the EAMS technician to rock the view in the HMD, since the view naturally rocks with the user's inherent postural tilt while standing.

[0079] In some embodiments, people looking at the 2D EAMS system screen on the display monitor 120 (e.g., the EAMS technician and other people in the room not wearing the HMD 110) can be more aware of what the user wearing the HMD 110 is doing. The processing system 100 receives the point of interest from the HMD 110 that the user is looking at and wobbles the EAMS view around that point on the EAMS 2D display. In some embodiments, the EAMS view focuses on and follows whatever point the HMD user is looking at to provide those people observing the medical procedure with more context about what the doctor is doing and the pattern of scanning information on the display and medical images. In other embodiments, the user's head direction (based on head position and orientation) or gaze direction is represented on the EAMS 2D display as a laser pointer or cursor. If the EAMS technician is using the cursor to highlight a location on the 2D EAMS display, the cursor may be indicated by the HMD 110 (e.g., as a point on the geometry or 3D line).

[0080] In some embodiments, the EAMS display indicates the battery or thermal status of the HMD 110 so that the physician's assistant can anticipate when the HMD 110 needs to be replaced with another one or have its batteries changed. Other notifications, warnings, or errors displayed by the HMD 110 can be mirrored on the EAMS display on the display monitor 120 to help the physician's assistant address those messages.

[0081] In various embodiments, bidirectional communication between the EAMS computing device 102 and the HMD 110 improves communication with a user who is remotely located, for example, rather than in the same operating room as the user wearing the HMD 110.

[0082] In some embodiments, data flowing from the HMD 110 to the EAMS computing device 102 is then routed (by the EAMS) to remote systems and users. This allows the remote user to better understand what the HMD user is doing and to provide supervision, assistance, feedback, or ask questions from a remote location. Data sent from the HMD 110 to the EAMS includes audio of the HMD user's voice, enabling, for example, remote voice control, note taking, and intercommunication between remotely located users. In some embodiments, the EAMS technicians themselves are located remotely from the operating room. In some embodiments, the HMD 110 can receive and send data to multiple EAMS simultaneously.

[0083] XI. Using Two-Way EAMS Communication to Reduce Computational Demands on the HMD Exemplary design factors for the HMD 110 include weight, size, heat dissipation, comfort, aesthetics, and battery life. Due to design tradeoffs such as reduced weight versus user comfort, the HMD 110 may have less computing power than the EAMS computing device 102, e.g., a high performance computer workstation that does not need to be lightweight. It would be advantageous to reduce the computing power required by the HMD 110 (e.g., by implementing more efficient algorithms) to further reduce weight, cost, and heat, and improve comfort and aesthetics.

[0084] In various embodiments, the HMD 110 sends information to the EAMS, including the position and orientation of the HMD 110, or the user's gaze direction. The HMD 110 or the EAMS can use any combination of the HMD position, HMD orientation, and user's gaze direction to determine the HMD 110's viewpoint (representing the user's viewpoint). Based on the viewpoint, the EAMS can pre-process the data to reduce the computational resources required by the HMD 110 to generate the resulting graphics (e.g., images or videos) on the HMD 110. In various embodiments, the EAMS sends the imaged patient's anatomical structure data to the HMD 110 (e.g., via the data computation device 104) as a 3D mesh of triangles or as a collection of 3D voxels (e.g., 3D textures). To accurately simulate occlusion and transparency (known to those skilled in the art as Painter's algorithm), the HMD 110 sorts these triangles or voxels so that those farther in spatial distance from the user are rendered first and those closest to the user are rendered last.

[0085] In some embodiments, the EAMS sorts the triangles and voxels before sending the sorted triangles and voxels to the HMD 110. Because the user may have physically moved while the EAMS was performing the sort (e.g., duration of less than one second), the viewpoint of the HMD 110 may have also changed during this duration. The user's movements during the EAMS sort are typically slight movements rather than significant changes in orientation or position, and therefore minimal artifacts occur when the HMD 110 renders the sorted triangles and voxels. In situations where the viewpoint changes more significantly during the EAMS sort, the HMD 110 may perform additional sorting operations to take into account the user's movements and mitigate any artifacts. Because the collection of triangles and voxels is further pre-sorted by the EAMS, reducing the sorting required by the HMD 110, performing the additional sorting operations requires fewer computational resources than a full sort by the HMD 110.

[0086] In some embodiments, instead of sending the anatomical structure shape and position of the catheter or medical instrument (known to those skilled in the art as remote rendering), the EAMS renders the imaging data and sends the rendered video image frames to the HMD 110. The video image frames include pixel color and depth from the user's eyes. This allows the HMD 110 to adjust the received frames to account for small changes in the HMD viewpoint that occur as a result of user movement. As previously mentioned, such user movement during image processing can cause artifacts in the rendered image. This process of adjusting old images is known to those skilled in the art as late reprojection or time warp.

[0087] XII. Using Two-Way EAMS Communication to Match Wireless Network Speeds In various embodiments, the HMD 110 worn by the user receives data from the EAMS computing device 102 via a wireless network (e.g., Internet, WIFI, Ultra Wideband, or Bluetooth). The capacity of the wireless network is limited and can change over time based on factors such as the number of other wireless devices nearby, the distance between the EAMS and the HMD 110, the placement of other equipment, walls, and people in the operating room, etc. The EAMS data can be delayed by network conditions, causing the position of the catheter or medical instrument displayed by the HMD 110 to lag behind their actual location within the patient, which in turn can make it more difficult for the physician to navigate the catheter to its intended target.

[0088] There is generally a tradeoff between the level of detail or fidelity in a computer-generated image and the amount of data that needs to be transmitted to reproduce that image at the HMD 110. In some embodiments, the HMD 110 can measure network receive strength, signal-to-noise level, latency, or throughput and feed this data back to the EAMS. Based on this feedback, the EAMS can form a closed-loop control system to adjust the level of detail in the data sent to the HMD 110. For example, when the network throughput is less than the rate at which the EAMS is sending data, the EAMS can throttle the data being sent in such a way that it is more likely to reach the HMD 110 in a timely manner.

[0089] In some embodiments, the EAMS computing device 102 sends the shape of the patient's anatomy to the HMD 110 (e.g., via the data computing device 104) as a 3D mesh of triangles or as a collection of 3D voxels. The EAMS may reduce the number of triangles, the precision of the triangle vertex locations, the resolution of the 3D textures, or any combination thereof. These actions result in a less detailed image of the anatomy displayed by the HMD 110. In some embodiments, the EAMS further reduces detail in areas of the anatomy that are at a greater distance from the HMD 110's viewpoint (known to those skilled in the art as foveated rendering) or that are at a greater distance from the location of a medical instrument, such as the distal end of a catheter.

[0090] In some embodiments, when the catheter is moving slowly, the EAMS sends data to the HMD 110 at a higher level of detail since transmission time is not a limiting factor. In contrast, when the catheter is moving quickly, the EAMS sends data to the HMD 110 at a lower level of detail but at a faster rate of transmission to ensure that the HMD 110 is updated in a timely manner to reflect the catheter movement.

[0091] In some embodiments, when the network speed is reduced, the EAMS reduces the level of detail of the catheter or medical instrument position and shape. For example, the EAMS reduces the position precision or reduces the number of control points in the spline. The EAMS allocates more detail or bandwidth to catheters that are moving fast (e.g., above a threshold rate of change or relative to other catheter motion) or have a distal end closer to the HMD gaze point. The EAMS can determine a larger image update rate for catheters that are moving faster (e.g., user-operated motion or system controlled). The EAMS can determine a larger image update rate for catheters that are located closer to the HMD gaze point.

[0092] XIII. Additional Configuration The disclosed embodiments are not mutually exclusive. They may be combined into one embodiment to provide a user with multiple options during a medical procedure. For example, a user may begin specifying a 3D spatial relationship according to a handheld controller embodiment, then put down the handheld controller and continue to refine the 3D spatial relationship according to a hands-free embodiment.

[0093] The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will recognize that numerous modifications and variations are possible in light of the foregoing disclosure.

[0094] Some portions of this description describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to convey the substance of their work to others skilled in the art. While described functionally, computationally, or logically, these operations will be understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Moreover, it has also proven convenient at times, without loss of generality, to refer to these arrangements of operations as modules. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combination thereof.

[0095] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In some embodiments, a software module is implemented in a computer program product that includes a non-transitory computer-readable storage medium that includes computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0096] Embodiments may also relate to products produced by the computational processes described herein. Such products may include information resulting from the computational processes, where the information is stored on a non-transitory, tangible computer-readable storage medium and may include any of the embodiments of the computer program products or other data combinations described herein.

[0097] Finally, the language used herein has been selected primarily for readability and instructional purposes; it should not be selected to delineate or limit the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but by any claims that issue on an application based thereon. Accordingly, the disclosure of the embodiments herein is intended to illustrate, but not limit, the scope of the invention, which is set forth in the appended claims.

Claims

1. a head mounted display (HMD) worn by a user; A display monitor; providing image data for display by the HMD based on data provided to the display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; providing user input information to the display monitor to update the user interface based on the user input and the position of the cursor; a processing system configured to A system comprising:

2. the processing system includes a frame grabber configured to receive the data to be provided to the display monitor for the user interface to be displayed by the display monitor. The system of claim 1 .

3. The processing system includes: A first computing device; a second computing device configured to provide the image data for display by the HMD and to receive the user input from the HMD; a signal generator configured to transmit information related to the user input to the first computing device; the first computing device generates the user input information using the information associated with the user input. The system of claim 1 .

4. the signal generator transmitting the information related to the user input as a USB signal; the first computing device generates the user input information without using input from a mouse or keyboard. The system of claim 3.

5. The second computing device receives the gaze direction of the user from the HMD; the first computing device providing the position of the cursor to the display monitor; The system of claim 3.

6. The HMD includes: Displaying an augmented reality graphic; and in response to determining that the user is interacting with the user interface, updating the display of the augmented reality graphic to track the position of the cursor; and further configured to: The system of claim 1 .

7. the processing system is further configured to determine that the action performed by the user is a selection of one of a plurality of user controls displayed in the user interface. The system of claim 1 .

8. The processing system is further configured to provide registration information to a 3D anatomical information source. The system of claim 1 .

9. providing image data for display by a head mounted display (HMD) worn by a user based on data provided to a display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; providing user input information to the display monitor to update the user interface based on the user input and the position of the cursor; A method comprising:

10. receiving the data to be provided to the display monitor for the user interface to be displayed by the display monitor; 10. The method of claim 9.

11. transmitting information related to the user input from the second computing device to the first computing device by a signal generator, the first computing device generating the user input information using the information related to the user input.

10. The method of claim 9.

12. the signal generator transmitting the information related to the user input as a USB signal; the first computing device generates the user input information without using input from a mouse or keyboard. The method of claim 11.

13. the second computing device receives the gaze direction of the user from the HMD, and the first computing device provides the position of the cursor to the display monitor. The method of claim 11.

14. displaying an augmented reality graphic by said HMD; in response to determining that the user is interacting with the user interface, updating the display of the augmented reality graphic to track a position of the cursor; Further comprising:

10. The method of claim 9.

15. determining that the action performed by the user is a selection of one of a plurality of user controls displayed in the user interface; 10. The method of claim 9.

16. providing the registration information to a 3D anatomical information source; 10. The method of claim 9.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, providing image data for display by a head mounted display (HMD) worn by a user based on data provided to a display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; providing user input information to the display monitor to update the user interface based on the user input and the position of the cursor; A non-transitory computer-readable storage medium that causes the one or more processors to

18. When executed by the one or more processors, receiving the data to be provided to the display monitor for the user interface to be displayed by the display monitor; and further storing instructions for causing the one or more processors to:

20. The non-transitory computer-readable storage medium of claim 17.

19. When executed by the one or more processors, transmitting information related to the user input from the second computing device to the first computing device by a signal generator, the first computing device using the information related to the user input to generate the user input information; and further storing instructions for causing the one or more processors to:

20. The non-transitory computer-readable storage medium of claim 17.

20. When executed by the one or more processors, Displaying an augmented reality graphic with the HMD; in response to determining that the user is interacting with the user interface, updating the display of the augmented reality graphic to track the position of the cursor; and further storing instructions for causing the one or more processors to:

20. The non-transitory computer-readable storage medium of claim 17.

Images