Bidirectional communication between head mounted display and electroanatomical system

Abstract

A processing system provides bidirectional communication between an electroanatomical navigation and mapping system (EAMS) and a head mounted display (HMD). A user wearing the HMD may interact with a user interface for EAMS information (e.g., 3D augmented reality graphics) displayed by the HMD, or EAMS information (e.g., 2D graphics) displayed by the display monitor, using hands-free or touch-free actions. For example, the processing system updates a position of a cursor on the display monitor to track a gaze direction of the user as determined by the HMD.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 333,900, filed on April 22, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to systems for electroanatomical mapping and manipulation of displays in a virtual reality environment, an augmented reality environment, or a mixed reality environment. Background Art

[0004] Electroanatomical navigation and mapping systems (EAMS) help physicians perform minimally invasive medical procedures. EAMS tracks the distal end of a medical tool (such as a catheter). EAMS displays the position of the medical tool relative to the patient's anatomy and other patient data via a real-time computer-generated image displayed on a (monoscopic) 2D computer screen (e.g., an LCD flat panel mounted on a cantilever above the patient).

[0005] Conventionally, users operate EAMS using a keyboard and mouse. Generally, the hands of a doctor performing a medical procedure are sterile and occupied by the task of holding and manipulating medical tools. Therefore, the doctor cannot interact directly with EAMS. Instead, another person (e.g., a technician) operates EAMS on behalf of the doctor. The doctor communicates requests to the EAMS technician via voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The disclosed embodiments have advantages and features that will be more readily apparent from the detailed description, the appended claims, and the accompanying drawings (or figures).A brief description of the drawings is as follows.

[0007] Figure 1 An example system environment for a processing system according to an embodiment is illustrated.

[0008] Figure 2 is a flow diagram of a process for user control in an augmented reality environment, according to an embodiment.

[0009] Figure 3 An example user interface in an augmented reality environment is illustrated according to an embodiment. Summary of the invention

[0010] The EAMS is integrated with a head mounted display (HMD) worn by the user (such as Sentiar TMIt is beneficial to supplement or connect with a head-mounted display (HMD) that tracks the position and orientation of the user's head and / or glasses / pupil. EAMS sends data about the patient's anatomical structure and the position and state of the catheter or other medical tools being tracked. The HMD generates a 3D image of the data from EAMS and displays the 3D image to the doctor. Users can rotate or otherwise change the view of their EAMS data by simply moving their heads, without verbally asking the EAMS technician to manipulate the view using a mouse or keyboard. In addition, the HMD can also present a user interface (e.g., menus and buttons) that can be controlled using a cursor, which is associated with the place where the user points to using his head or eyes / pupil. This allows further interaction with the received EAMS data, such as zooming in and out, and turning on or off various visual elements. The benefits of the embodiments described herein include: providing users with a better understanding of complex three-dimensional spatial relationships in EAMS data, higher accuracy in positioning catheters or other medical tools, and a reduced need for verbal communication with EAMS technicians, as well as other advantages.

[0011] The HMD provides the above benefits by receiving data from the EAMS and allowing the user to view the EAMS data and manipulate the view of the data. In conventional systems, the HMD user cannot affect the state or data of the EAMS itself. If the user wants to change the data on the EAMS itself (e.g., edit the shape of the patient's anatomy), the user must verbally request the technician to perform the operation on his or her behalf. Verbally communicating complex spatial data using commands is difficult and tedious, and there is a risk that the technician will misunderstand some nuances in the user's communication.

[0012] The embodiments herein describe novel systems and methods that utilize bidirectional communication between an HMD and an EAMS, and the benefits of doing so, for example, as compared to conventional systems that use one-way communication from the EAMS to the HMD.

[0013] In various embodiments, a system includes a head mounted display (HMD) worn by a user; a display monitor; and a processing system configured to: provide image data for display by the HMD based on data provided to the display monitor for a user interface displayed by the display monitor; receive a gaze direction of a user from the HMD; determine a position of a cursor displayed on the user interface based on the gaze direction of the HMD user; provide the position of the cursor to the display monitor to update the display of the cursor on the display monitor; receive user input from the HMD in response to an action performed by the user; and provide user input information to the display monitor to update the user interface based on the user input and the position of the cursor.

[0014] In some embodiments, a processing system includes a frame grabber configured to receive data provided to a display monitor for a user interface displayed by the display monitor.

[0015] In some embodiments, the processing system includes: a first computing device; a second computing device configured to provide image data for display by the HMD and to receive user input from the HMD; and a signal generator configured to send information associated with the user input to the first computing device, wherein the first computing device uses the information associated with the user input to generate user input information.

[0016] In some embodiments, the signal generator sends information associated with the user input as a USB signal, and wherein the first computing device generates the user input information without using input from a mouse or keyboard.

[0017] In some embodiments, the second computing device receives a gaze direction from a user of the HMD, and wherein the first computing device provides the position of the cursor to the display monitor.

[0018] In some embodiments, the HMD is further configured to: display augmented reality graphics; and in response to determining that the user is interacting with the user interface, update the display of the augmented reality graphics to track the position of the cursor.

[0019] In some embodiments, the processing system is further configured to determine that the action performed by the user is a selection of a user control among a plurality of user controls displayed in the user interface.

[0020] In some embodiments, the processing system is further configured to provide registration information to the 3D anatomical information source.

[0021] In various embodiments, a method includes: providing image data for display by an HMD based on data provided to a display monitor for a user interface displayed by the display monitor; receiving a gaze direction from a user of the HMD; determining a position of a cursor displayed on the user interface based on the gaze direction of the HMD user; providing the position of the cursor to the display monitor to update display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; and providing user input information to the display monitor to update the user interface based on the user input and the position of the cursor.

[0022] In some embodiments, the method further includes receiving data provided to a display monitor for a user interface displayed by the display monitor.

[0023] In some embodiments, the method further includes sending, by the signal generator, information associated with the user input from the second computing device to the first computing device, wherein the first computing device generates the user input information using the information associated with the user input.

[0024] In some embodiments, the signal generator sends information associated with the user input as a USB signal, and wherein the first computing device generates the user input information without using input from a mouse or keyboard.

[0025] In some embodiments, the second computing device receives a gaze direction from a user of the HMD, and wherein the first computing device provides the position of the cursor to the display monitor.

[0026] In some embodiments, the method further includes displaying, by the HMD, the augmented reality graphics; and in response to determining that the user is interacting with the user interface, updating the display of the augmented reality graphics to track the position of the cursor.

[0027] In some embodiments, the method further includes determining that the action performed by the user is a selection of a user control among a plurality of user controls displayed in the user interface.

[0028] In some embodiments, the method further comprises providing registration information to the 3D anatomical information source.

[0029] In various embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform any of the steps of the methods described herein.

[0030] In some embodiments, a processing system including an HMD sends live voice to EAMS for intercom with a remote user, dictation of annotations and process notes, or voice control of EAMS functions.

[0031] In some embodiments, the processing system including the HMD can remotely operate functions that would otherwise be provided to the EAMS technician via a keyboard and mouse. Example functions include: causing EAMS to start and stop collecting catheter-tissue contact points to generate or update the geometry of the patient's anatomy; switching which catheter / tool ​​to use for mapping; marking a particular frame of an ultrasound image or fluoroscopic image to be saved for later review; annotating features in the saved ultrasound image or fluoroscopic image; changing the level of detail at which EAMS generates the patient's anatomy geometry; specifying to EAMS that a sub-region of the anatomy is to be selected for an operation (such as cutting, trimming, deleting, smoothing, pushing inwards, or pushing outwards); commanding EAMS to Perform operations on selected sub-regions of the patient's anatomy (such as cutting, trimming, deleting, smoothing, pushing inward or outward); specify to EAMS how the current anatomical geometry or electrical map should be merged with other anatomical geometry or electrical maps previously acquired (including anatomical geometry or electrical maps acquired from ultrasound images, CT images, and MRI images); command EAMS to turn on / off the collection of electrograms (EGMs); specify to EAMS which electrodes on the catheter to highlight or turn on / off; specify to EAMS which electrodes it should display on its 2D screen The system can specify to EAMS the type or style of visualization and which type or style of visualization it should send to the HMD headset; specify to EAMS the thresholds and endpoints for the mapping from data to color; command EAMS to start and stop animating visualizations; specify to EAMS to create marker points in space and / or time and attach annotations or comments to the marker points for subsequent editing or review (the points can be about patient anatomy, catheter to tool tip location, location in an ultrasound image, CT image, MRI image, or X-ray image, or signal features in an electrogram); specify to EAMS to create marker points in space and / or time and attach annotations or comments to the marker points for subsequent editing or review AMS specifies which anatomical samples, marks, labels, or landmarks are selected, edited, or deleted; commands EAMS to take a snapshot image or video of the current EAMS screen, data for subsequent playback review or export to a patient record; commands EAMS to measure the distance between a midpoint of an anatomical structure and a feature (virtual caliper); commands EAMS to measure the timing difference between timestamps and features in an electrogram waveform; commands EAMS to rewind, fast forward, pause, and stop playback of previously recorded data; and commands EAMS to create annotations, such as lines and curves projected onto the 3D surface of the anatomical structure.

[0032] In some embodiments, the position and orientation of the user's head and / or eye gaze is sent by the HMD to the EAMS for the following purposes: displaying what the user is looking at on the EAMS 2D display monitor (in the HMD); sharing what the user is looking at with a remote EAMS and other users wearing the HMD; recording what the user is looking at in a record of medical procedure data; or moving computing tasks and workloads from the HMD to the EAMS computing device (e.g., remote rendering with time warping or post-projection, or reordering voxels and / or polygons by distance from the viewing user's location).

[0033] In some embodiments, the battery and / or temperature status of the HMD is sent to and displayed by the EAMS to help the user prepare for swapping to another HMD battery and troubleshooting.

[0034] In some embodiments, the processing system measures the throughput, jitter, latency, signal-to-noise ratio, and / or loss rate of the wireless network between the EAMS and the HMD so that the EAMS can reduce or increase the amount of data sent to the EAMS to manage the delay between when the EAMS sends the data and when the HMD displays the data. The processing system can implement mesh extraction, progressive refinement of 2D images or 3D images, or foveated rendering, where the highest fidelity part of the image is where the user is looking or where the catheter tip (or another part of the medical tool) is moving. DETAILED DESCRIPTION

[0035] As disclosed herein, embodiments of two-way EAMS communications provide a user wearing an HMD with greater and improved control over a user interface or other functionality.

[0036] In some embodiments, the HMD provides a user interface (eg, buttons and menus) that can be controlled hands-free by a user (such as a physician) and allows the physician to operate functions of the EAMS that would otherwise require the physician to make a verbal request to an EAMS technician.

[0037] Example commands that the HMD may send to the EAMS include: starting or stopping the collection of contact points between a medical tool and tissue to generate or update information (e.g., geometry or shape) of the patient's anatomy; marking a particular frame of an ultrasound image or fluoroscopic image to be saved for later review; annotating features in the saved ultrasound image or fluoroscopic image; switching which medical tool is used for mapping; turning on or off the collection of electrograms (EGMs); performing operations (such as cutting, trimming, deleting, smoothing, pushing inwards or outwards) on a selected sub-region of the patient's anatomy; starting or stopping the display of an animated visualization (e.g., a BWI CARTO ripple map); taking a snapshot image or video of the current EAMS screen, data for later playback review or export to a patient's medical record; measuring the distance between a point in the anatomy and a feature (e.g., a virtual caliper); measuring the timing difference between a timestamp and a feature in an electrogram waveform; and requesting rewind, fast forward, pause, and stop playback of previously recorded data.

[0038] The HMD may send information to the EAMS for updating the display of the HMD. Example information includes: the level of detail at which EAMS generates the patient's anatomical geometry (e.g., changing to a more simplified or more detailed level); sub-regions of the anatomy that are to be selected for manipulation (such as cutting, trimming, deleting, smoothing, pushing inwards or outwards); instructions to merge the anatomical geometry or electrogram with previously acquired data (e.g., data acquired from an ultrasound image, a CT image, or an MRI image); which electrodes on the catheter are highlighted or turned on / off; which type or style of visualization (e.g., a map) should be displayed on a 2D display monitor or HMD (e.g., in 3D); thresholds and endpoints for mapping from data to color; creation of marker points in space or time, and associated annotations or comments for subsequent editing or review (the marker points can be about the patient's anatomy); catheter to tool tip position; medical tool position in an ultrasound image, CT image, MRI image, or X-ray image; signal characteristics in an electrogram; and which anatomical samples, markers, labels, or landmarks are selected, edited, or deleted.

[0039] In various embodiments, functions performed by the EAMS technician using the mouse and keyboard may also be controlled by the user wearing the HMD.

[0040] I. Electroanatomical Navigation and Mapping System (EAMS)

[0041] In an electrophysiology procedure, an operator-user and / or electrophysiologist-user can use a 2D display monitor, mouse, and keyboard to specify 3-dimensional spatial relationships with 3 degrees of freedom or 6 degrees of freedom (e.g., 3D position, direction, and magnitude, or 3D position and 3D orientation in 3D space), and any number of supporting computer systems that provide patient data. An example implementation is EAMS. Any number of supporting systems can supplement the EAMS data with additional procedure data or instrument controls adjacent to or within the EAMS data.

[0042] In some programs, the user imports a 3D model of the patient's anatomy (derived from a preoperative CT, MRI, or ultrasound scan, or from a cardiac mapping procedure previously performed using EAMS). The user then specifies how the imported 3D model relates to the current intraoperative 3D model of the heart in space. Typically, the user must identify three or more corresponding 3D points on the surface of the model. EAMS then calculates the best fit transformation between the two models. In other cases, the user uses the mouse to interactively apply any number of translation, rotation, and / or scaling operations to the model using a 2D display monitor, mouse, and / or keyboard.

[0043] In some procedures, the catheter is partially or fully automatically steered via any number of mechanical cables, motors, or magnetic or pneumatic actuators inside or outside the catheter. The user specifies the 3D trajectory vector, 3D path, and / or 3D target destination location. The automatic system then calculates the appropriate actuation to steer and drive the catheter as the user has specified.

[0044] In some ablation systems, the direction in which the ablation energy is directed is electronically or mechanically steerable (eg, high intensity focused ultrasound, phased array RF antennas). The user specifies a 3D vector in which the ablation energy is directed from a launch point on the catheter.

[0045] In the above example, because the EAMS user is using a 2D display monitor and a mouse with two degrees of motion, the user can only observe and adjust two degrees of freedom of the desired 3D spatial relationship (which can have 3 degrees of freedom to 6 degrees of freedom) at a time. The user frequently cycles between (1) adjusting translation in 2D, (2) adjusting rotation in 2D, and (3) changing viewing orientation (i.e., pitch and yaw) in 2D. The user will need to repeatedly iterate between the three modes to interpret, construct, adjust, and check the work-in-progress 3D spatial relationship until it sufficiently resembles the user's intent. This manipulation requires practice and skill, and can be tedious, time consuming, and error-prone.

[0046] II. System Overview

[0047] Figure 1 An example system environment for a processing system 100 according to an embodiment is illustrated. The system environment includes the processing system 100, one or more HMDs 110, one or more display monitors 120, and one or more input devices 130. The processing system 100 includes an EAMS computing device 102 (also referred to herein as "EAMS"), a data computing device 104, a frame grabber 106, and a signal generator 108. In other embodiments, the functions of the processing system 100 can be performed by any number of devices. For example, instead of an EAMS computing device 102 that is separate from the data computing device 104, the processing system 100 includes a computing device (including a memory and one or more processors) that performs the functions of both the EAMS computing device 102 and the data computing device 104, as well as the functions of the frame grabber 106 and the signal generator 108.

[0048] In some embodiments, the HMD 110 and EAMS computing device 102 functions are executed on separate devices. One computing device is part of EAMS and the other computing device is part of HMD 110. In other embodiments, the functions of the two computing devices are combined into a single computing device. In other embodiments, some functions of the EAMS computing device 102 or HMD 110 are executed on a remotely located server, which may also be part of the processing system 100.

[0049] The EAMS computing device 102 is communicatively coupled to the data computing device 104 (e.g., via the Internet or another form of wireless connection or a wired connection). The EAMS computing device 102 is communicatively coupled to one or more input devices 130 (coupled to, for example, a keyboard, a mouse, or a source of 3D anatomical information). The EAMS computing device 102 sends information for display on a display monitor 120 (e.g., a computer monitor having a display such as an LCD, LED, OLED, plasma, or touch screen display). The frame grabber 106 receives the information sent to the display monitor 120 and sends the information to the data computing device 104.

[0050] The data computing device 104 is communicatively coupled to one or more HMDs 110 via a wireless connection or a wired connection. In conventional systems, the EAMS computing device 102 receives input from an input device (such as a mouse or keyboard). The signal generator 108 can provide input to the EAMS computing device 102 similar to the input from the mouse or keyboard. The signal generator 108 uses the information from the data computing device 104 to generate an input signal (e.g., a USB signal). Figure 2 and Figure 3Describe an example use case.

[0051] Figure 2 is a flow chart of a process 200 for user control in an augmented reality environment, according to an embodiment. Figure 3 An example user interface in an augmented reality environment according to an embodiment is illustrated. Processing system 100 performs process 200, which can be used in cardiac medical procedures, such as Figure 3 This is illustrated by the example user interface shown in In other embodiments, process 200 is applicable to other types of medical procedures.

[0052] In step 210, the processing system 100 provides image data for display by the HMD 110 based on data provided to the display monitor 120 for a user interface displayed by the display monitor 120. In some embodiments, the data computing device 104 may perform step 210 and the EAMS computing device 102 may provide data to the display monitor 120 for the user interface. The data computing device 104 receives the data provided to the display monitor 120 via the frame grabber 106. Figure 3 , the HMD 110 displays an augmented reality graphic 300 that is a model of a patient's heart. The HMD 110 also displays other augmented reality graphics including a catheter 320 and user controls 330. Since the user is looking toward the display monitor 310, the HMD 110 displays the augmented reality graphic 300 and other graphics as holograms overlaid on the display monitor 310 from the user's perspective.

[0053] In step 220, processing system 100 receives a gaze direction from a user of HMD 110. HMD 110 determines the gaze direction using sensor data captured by one or more sensors of HMD 110. For example, the sensor data represents head direction or eye / pupil direction.

[0054] In step 230, the processing system 100 determines the position of the cursor displayed on the user interface based on the gaze direction of the HMD user. Figure 3 As shown in , the display monitor 310 displays a cursor 350 (e.g., a mouse cursor) on the user interface. Additionally, the HMD 110 displays a marker 340 to indicate the gaze direction of the HMD user. The processing system 100 can control the position of the cursor 350 based on the position of the marker 340. Specifically, in step 240, the processing system 100 provides the position of the cursor 350 to the display monitor 310 to update the display of the cursor 350 on the display monitor 310. In response to determining that the marker 340 is located within the area of ​​the user interface displayed by the display monitor 310, the processing system 100 updates the position of the cursor 350 to track the position of the marker 340.

[0055] In step 250, the processing system 100 receives user input from the HMD 110 in response to an action performed by the user. The user input may be a hands-free input such as an eye gesture (e.g., winking or winking), a head gesture (e.g., nodding), or a voice command. The processing system 100 may determine that the user input is an interaction with one or more user controls (e.g., buttons, sliders, menus, text input boxes) displayed in a user interface.

[0056] In step 260, the processing system 100 provides user input information to the display monitor 310 to update the user interface based on the user input and the position of the cursor 350. As an example, the user performs a user input to start or stop a mapping or registration procedure. Instead of using a mouse or keyboard to interact with the user interface on the display monitor, the user can interact with the user interface by directing the user's gaze to a virtual button (e.g., an augmented reality graphic) displayed by the HMD 110, or to a button on the display monitor (where the cursor tracks the user's gaze direction, such as by Figure 3 102 ), using hands-free input to start or stop the program. Instead of using a mouse to click or press a keyboard key, the user provides hands-free input such as a wink or a blink to indicate a selection of a virtual button or a button on a display monitor. The signal generator 108 sends information associated with the user input from the HMD 110 to the EAMS computing device 102, which processes the information as if it were input from a mouse or keyboard received via a USB signal.

[0057] As another use case, the user performs a user input to store the current 3D position of the catheter tip as a marker or sample point. The data computing device 104 continuously calculates the 2D coordinates in the coordinate frame of the user interface of the display monitor 120, where the HMD looks at the marker (e.g., Figure 3 The marker 340 shown in FIG. 1 overlaps the user interface from the perspective of the user wearing the HMD. The data computing device 104 then generates a USB mouse signal to update the cursor on the display monitor 120 to the coordinates so that the cursor appears to follow the HMD gaze marker. The user navigates the cursor to a drop-down menu of the user interface and selects the drop-down menu by winking or performing another type of touch-free action. The user can again use the gaze direction and another action to select an option from the menu, such as "store sample point."

[0058] In various embodiments, the EAMS and HMD can send data via wired or wireless transmission (including USB, Ethernet, PCI, Bluetooth, Wi-Fi, ultra-wideband RF, optical links, and with topologies such as direct point-to-point, token ring, mesh), or via a central router. Examples of HMD devices include optical see-through augmented reality (AR) headsets (e.g., Microsoft HoloLens, ODG R-7, Magic Leap 1 and Magic Leap 2, or Nreal Light), camera-based see-through AR headsets (e.g., Oculus Quest, Varjo XR-3, or Lynx R1), fully opaque virtual reality (VR) headsets (e.g., HTC Vive and Valve Index), and other displays that can track the position or orientation of the user's head or the direction of gaze, and displays from the user's perspective.

[0059] Although the embodiments described herein refer to a head mounted display (HMD) 110, the disclosed systems and methods may also be applied to a non-head mounted display (e.g., fixed to a floor, ceiling, wall, table, or cart) that is monoscopic or stereoscopic (e.g., Looking Glass 8k Gen 2, Sony LMD-X310MT, or Steris VividImage 4k surgical display) and coupled with a head or eye tracker (e.g., Tobii Pro Fusion, Tobii Pro Spectrum, or Tobii Tracker 5). In some embodiments, the system combines a head mounted eye tracking device (e.g., Tobii Pro Glasses 3, Pupil Labs Core, ArgusScience, or ETVision) with a non-head mounted display.

[0060] In various embodiments, the user interface for capturing user commands for the EAMS may use any combination of sensing modalities including head position and orientation, eye or pupil gaze direction, audio and voice input, hand or finger tracking, EEG (electroencephalography) or EMG (electromyography) sensors, facial gesture tracking (e.g., via a camera, depth sensor, ultra-wideband RF, or short-range radar), reed switches, a handheld remote controller, a laser pointer, a foot pedal, and a button or joystick mounted on a display or the proximal end of a catheter or medical tool.

[0061] Examples of EAMS systems include the Johnson & Johnson Biosense Webster CARTO TM , St. Jude EnsiteTM Velocity TM , Medtronic LocaLisa TM , Medtronic CardioInsight, Medtronic Affera.

[0062] In various embodiments, ultrasound images processed by an EAMS or HMD are acquired by an intracardiac ultrasound transducer, a transesophageal ultrasound transducer, an external ultrasound transducer, a 2D ultrasound transducer, or a 3D ultrasound transducer, with or without ultrasound contrast agents.

[0063] In various embodiments, the EAMS includes one or more navigation systems, such as navigation systems for procedures involving one or more of the following: heart, brain, sinus, jaw, dental, spine, knee, hip, lung, prostate, liver, kidney, vascular, prostate, pancreas, uterus, and abdomen.

[0064] The embodiments described herein may be used in conjunction with any number of catheters or other medical tools that are manually manipulated by a user's hand or assisted or controlled by a motor or other mechanical actuator (e.g., automatically controlled or power-assisted medical tools).

[0065] The disclosure herein generally refers to users as doctors. However, the disclosed embodiments may also be applied to other types of users, including veterinarians, nurses, trainees, instructors, technicians, engineers, administrators, and demonstrators, as well as other types of health care providers.

[0066] III. Head-mounted display (HMD)

[0067] In various embodiments, the HMD 110 includes one or more displays and user input devices. In some embodiments, user input is achieved through one or more microphones. In some embodiments, user input is achieved by observing the position and posture of the hands (e.g., using a camera or neuromuscular sensing). In some embodiments, user input is achieved by the relative position and posture of the HMD (e.g., gaze or head position cursor). In some embodiments, user input is achieved by a communicatively coupled sensor to provide any number of button inputs, hand positions, postures, or neuromuscular signaling. In some embodiments, user input is achieved by tracking the position and posture of the eyes (e.g., camera, neuromuscular sensing).

[0068] When viewed in a stereo head tracking mixed reality HMD, the ongoing 3D spatial relationship can be more prominent to the user than when shown on a 2D display. In addition, the ongoing 3D spatial relationship can be adjusted simultaneously as it is examined without the user having to switch modes.

[0069] IV. 3D Registration

[0070] In an embodiment, a system including an HMD 110 worn by a user, a 2D display monitor 120, and a processing system 100 is configured to provide any number of 3D anatomical information and respond to transformations between 3D anatomical coordinate systems, enabling a user to intuitively specify, update, and / or observe the registration of the 3D anatomical information in the combined coordinate system.

[0071] In some embodiments, 3D anatomical information (e.g., registration information) is generated and sent by EAMS intraoperatively and sent to processing system 100 for registration with another 3D anatomical information source (e.g., input device 130). In other embodiments, 3D anatomical information is sent through a picture archiving and communication system (PACS) for registration. In other embodiments, 3D anatomical information is generated by processing system 100 by combining 3D information from one source (e.g., catheter position) with another source (e.g., electrogram recording system, ultrasound).

[0072] In some embodiments, the processing system 100 sends registration information to the 3D anatomical information source by simulating inputs native to the 3D anatomical information source to assist in registration (e.g., a sequence of any number of keyboard operations or mouse operations). These operations will apply the normal registration operations of the existing 3D anatomical information source. In other embodiments, the registration information is communicated from the processing system 100 to the 3D anatomical information source by providing a set of corresponding points from the source information source to the destination information source. In some embodiments, the registration information is communicated from the processing system 100 to the 3D anatomical information source by providing the ability to transform any input position and / or posture from one anatomical information source to another anatomical information source. In some embodiments, the transformation is separately included and displayed within the processing system 100, and the information is converted within the processing system 100, and the communication between the anatomical information sources is communicated within each system native coordinate system.

[0073] V. Catheter Manipulation

[0074] In an embodiment, a system including an HMD 110 worn by a user and a processing system 100 enables the user to intuitively specify, update, and / or observe manipulation of any number of catheters, the processing system 100 being configured to receive at least one source of 3D anatomical information and to communicate with at least one catheter manipulator.

[0075] In some embodiments, the user specifies a target 3D position for the catheter manipulator. In some embodiments, the user specifies a sequence of target 3D positions for the catheter manipulator to create a target path. In some embodiments, the user specifies a 3D position and a target pose for the HMD manipulator to specify both the position and orientation of the catheter at any number of positions. In some embodiments, the processing system 100 communicates with the manipulator via EAMS. In some embodiments, the processing system 100 communicates with the catheter manipulator independently of EAMS.

[0076] VI. Controllable catheter

[0077] In an embodiment, a system including an HMD 110 worn by a user and a processing system 100 enables a user to intuitively specify, update, and / or observe 3D parameters of a catheter ablation system, and the processing system 100 is configured to receive at least one 3D anatomical information source and communicate with at least one configurable catheter ablation system. In some embodiments, the user specifies and observes direction detection parameters (e.g., electrode selection in an array, ultrasound). In some embodiments, the user specifies ablation direction parameters (e.g., electrodes in an array, phase, laser direction). In some embodiments, the processing system 100 communicates with the catheter ablation system via EAMS. In some embodiments, the processing system 100 communicates with the catheter ablation system independently of EAMS.

[0078] VII. User Input

[0079] In some embodiments, the user input from HMD 110 is implemented by detecting and sending any number of click operations indicating discrete events (e.g., start, stop). In some embodiments, the click operation is implemented by keeping the user's head posture within the movement tolerance for a period of time (gaze stay). In other embodiments, the click operation is implemented by a handheld controller that communicates with HMD 110 and / or processing system 100. In some embodiments, the operation is detected by processing the voice command using a dictionary, or by using the derived intention of any number of voice intention detection operations. In some embodiments, the operation is detected by physical gestures including facial expressions, hand gestures or movements, eye gestures or movements, etc. Physical gestures can be detected by observing neuromuscular signals and are classified as operations. Additionally, physical gestures can be detected by processing a sequence of image data from a 2D sensor (including electro-optical, or infrared (IR)). In some embodiments, these discrete inputs are segmented into a range of variable inputs to provide a range of intermediate values ​​(e.g., [0-10]).

[0080] In various embodiments, 2D user input from HMD 110 is implemented by detecting and sending absolute or relative 2D position changes. In some embodiments, this 2D information is calculated by processing the position of the HMD gaze cursor relative to the extent of a virtual or physical display. In some embodiments, coordinates can be transformed between a virtual display and a physical display to facilitate manipulation by a user wearing HMD 110, for example, an electrogram can be increased in virtual scale to provide greater precision with hands-free control as in EGM annotations.

[0081] In various embodiments, 3D user input from HMD 110 is implemented by detecting and sending absolute or relative 3D position changes. In some embodiments, the 3D information is calculated by processing the 3D position changes of the user wearing HMD 110. In some embodiments, 3D information is calculated by processing the position of the user's single hand input. In some embodiments, 3D information is calculated by processing the difference between the two hand inputs of the user. Hand input can be implemented by camera-based hand tracking or handheld or hand-worn controller input. In various embodiments, in addition to 3D position, 3D position and posture information is available. The processing system 100 can calculate this information based on a single input (e.g., the position and posture of a single hand) or based on a combination of inputs (e.g., the position of one hand forming the basis of a posture vector, and the relative position of the other hand).

[0082] VIII. Multi-user use case

[0083] The above embodiments may also be used by multiple users simultaneously or sequentially to allow a collection of users to collaborate in specifying 3D spatial relationships. For example, one user wearing an HMD may use a handheld controller to make large, rough adjustments to the ongoing 3D spatial relationship, and then another user wearing another HMD (who may have sterile hands) may make small, precise refinements to the ongoing 3D spatial relationship. Similarly, an EAMS operator sitting at a desk may assist the user wearing the HMD by using a mouse and keyboard to make large, rough adjustments to the ongoing 3D spatial relationship, while the user wearing the HMD standing next to the patient may make smaller, precise refinements to the ongoing 3D spatial relationship.

[0084] In some embodiments, when multiple users are given the ability to specify 3D spatial relationships, if the system enforces that only a single user is allowed to virtually grasp a pre-operative model, 3D trajectory, 3D direction and magnitude, or 3D point at a time, the user may more easily understand how to operate the processing system 100. Depending on whether the processing system 100 is composed of a single or multiple processing units, a conventional shared memory mutual exclusion or untrusted mutual exclusion algorithm (e.g., Suzuki-Kasami algorithm, Ricart-Agrawala algorithm) may be used to block a user from grasping a pre-operative model, 3D trajectory, 3D direction and magnitude, or 3D point virtually grasped by the current owner until the current owner has released it.

[0085] In some embodiments, HMD 110 must send the 3D spatial relationship back to EAMS computing device 102. In some embodiments, HMD 110 keeps the ongoing 3D spatial relationship private to itself until the user signals that they are finished specifying the 3D spatial relationship, whereupon HMD 110 sends the completed 3D spatial relationship to EAMS computing device 102. In other embodiments, HMD 110 continuously or periodically sends the ongoing 3D spatial relationship to EAMS computing device 102 while the user is still manipulating and refining the ongoing 3D spatial relationship.

[0086] IX.EAMS API

[0087] In some embodiments, the EAMS API outputs the position, orientation and shape of the catheter, the geometry and visualization description of the 3D heart model, and / or the waveform of the signal sensed by the electrode on the catheter. In addition to outputting these elements through the API, EAMS usually displays the element on its own 2D display. HMD 110 can use the data from the EAMS API to reconstruct the appearance of the part of the EAMS display monitor 120. This allows HMD 110 to perform pattern matching between the live video obtained by the camera of the HMD 110 (with the view of the EAMS 2D display monitor) and the reconstructed EAMS element (rather than the desktop video captured by the frame grabber 106). This has the benefit of requiring fewer hardware components.

[0088] In some embodiments, the waveform of the signal captured by the catheter electrode can not only appear on the EAMS display monitor 120, but can also be digitized by a device external to EAMS (e.g., GE Cardiolab). The external digitizer device (input device 130) can send the waveform of the signal to the HMD 110 via the processing system 100 using a digitizer API independent of the EAMS API.

[0089] In some embodiments, the EAMS computing device 102 assists the HMD 110 in detecting EAMS video captured in the camera video of the HMD 110 by adding one or more additional 2D graphical elements to the EAMS screen shown by the display monitor 120 and sending a description of the additional 2D graphical elements through the EAMS API. The additional elements are designed to be easier to identify in the camera video of the HMD than typical EAMS desktop video (by being high contrast and having unique features). Examples of additional 2D graphical elements include Aruco fiducials, QR codes, April tags, and 2D barcodes.

[0090] X. Use of two-way EAMS communication for improved collaboration with users viewing a 2D EAMS display

[0091] Because conventional EAMS systems without an HMD use a 2D display, the EAMS technician will slightly rotate the view back and forth (swing) around the point of interest to give the viewer a better understanding of the displayed 3D spatial relationships. This is often done, for example, when a physician is using a catheter tip to apply radio frequency (RF) energy to a specific point in a patient's tissue and must carefully control the position of the tip in the anatomy. Because the HMD is stereoscopic, the EAMS technician has less need to swing the view in the HMD, and therefore the view swings naturally with the inherent postural sway of the user when standing.

[0092] In some embodiments, a person (e.g., an EAMS technician and other people in the room who are not wearing HMD 110) who is viewing the 2D EAMS system screen on the display monitor 120 can be given more knowledge of what the user wearing HMD 110 is doing. The processing system 100 receives a point of interest that the user of the HMD 110 is viewing and swings the EAMS view about that point on the EAMS 2D display. In some embodiments, the EAMS view is centered on and follows any point that the HMD user is viewing to provide those who are viewing the medical procedure with more context about what the doctor is doing, as well as the pattern of information and medical images that they are scanning on the display. In other embodiments, the user's head direction (based on head position and orientation) or gaze direction is represented as a laser pointer or cursor on the EAMS 2D display. If the EAMS technician uses a cursor to highlight a location on the 2D EAMS display, the cursor can be shown by the HMD 110 (e.g., as a point on a geometric structure or a 3D ray).

[0093] In some embodiments, the EAMS display shows the battery status or thermal status of the HMD 110 so that the physician's assistant can anticipate when the HMD 110 will need to be swapped with another HMD, or the battery of the HMD 110 replaced. Other notifications, warnings, or errors displayed by the HMD 110 may be mirrored on the EAMS display on the display monitor 120 so that the physician's assistant can help address those messages.

[0094] In various embodiments, two-way communication between the EAMS computing device 102 and the HMD 110 improves communication with a user who is remotely located (eg, not in the same procedure room as the user wearing the HMD 110 ).

[0095] In some embodiments, data flowing from the HMD 110 to the EAMS computing device 102 is then routed (by the EAMS) to the remote system and user. This allows the remote user to better understand what the HMD user is doing and provide supervision, assistance, feedback, or ask questions from a remote location. The data sent from the HMD 110 to the EAMS includes, for example, audio of the HMD user's voice, thereby allowing remote voice control, annotation recording, and intercom between remotely located users. In some embodiments, the EAMS technician himself is located away from the operating room. In some embodiments, the HMD 110 can receive data from multiple EAMS and send data to multiple EAMS simultaneously.

[0096] XI. Using two-way EAMS communication to reduce the computational requirements of the HMD

[0097] Example design factors for the HMD 110 include weight, size, heat dissipation, comfort, aesthetics, and battery life. Due to design trade-offs such as reducing weight for user comfort, the HMD 110 can have less computing power than the EAMS computing device 102 (e.g., no lightweight high-performance computer workstation is required). It is advantageous to reduce the computing power required by the HMD 110 (e.g., by implementing more efficient algorithms) to further reduce weight, cost, and heat, and improve comfort and aesthetics.

[0098] In various embodiments, the HMD 110 sends information to the EAMS, including the position and orientation of the HMD 110, or the gaze direction of the user. The HMD 110 or the EAMS can use any combination of the HMD position, the HMD orientation, and the gaze direction of the user to determine the perspective of the HMD 110 (representing the perspective of the user). Based on the perspective, the EAMS can pre-process the data to reduce the computing resources required by the HMD 110 to generate the resulting graphics (e.g., images or videos) on the HMD 110. In various embodiments, the EAMS sends the imaged patient anatomical structure data (e.g., via the data computing device 104) as a 3D mesh of triangles or as a collection of 3D voxels (e.g., 3D textures) to the HMD 110. The HMD 110 sorts these triangles or voxels so that those triangles or voxels that are farther away from the user in spatial distance are rendered first, and those triangles or voxels that are closest to the user are rendered last to correctly simulate occlusion and transparency (referred to as the painter's algorithm by those skilled in the art).

[0099] In some embodiments, EAMS sorts the triangles and voxels before sending the sorted triangles and voxels to HMD 110. Since the user may have physically moved (e.g., for a duration of less than one second) while EAMS is performing the sorting, the viewing angle of HMD 110 may also have changed during that duration. The user's movement during EAMS sorting is typically slight movement, rather than a significant change in orientation or position, so minimal artifacts appear when HMD 110 renders the sorted triangles and voxels. In situations where the viewing angle changes to a larger range during EAMS sorting, HMD 110 may perform additional sorting operations to account for the user's movement and mitigate any artifacts. Performing additional sorting operations requires fewer computing resources than a full sort by HMD 110, because the collection of triangles and voxels is still pre-sorted by EAMS, which reduces the sorting required by HMD 110.

[0100] In some embodiments, EAMS renders the imaging data and sends the rendered video image frames to HMD 110 instead of sending the anatomical shape and the position of the catheter or medical tool (referred to as remote rendering by those skilled in the art). The video image frames include the color of the pixels and the depth from the user's eyes. This enables HMD 110 to adjust the received frames to account for slight changes in the HMD's viewing angle caused by the user's movement. As previously described, such user movement during image processing may cause artifacts in the rendered image. This process of adjusting outdated images is referred to as late reprojection or time warping by those skilled in the art.

[0101] XII. Using two-way EAMS communication to adapt to wireless network speeds

[0102] In various embodiments, the HMD 110 worn by the user receives data from the EAMS computing device 102 via a wireless network (e.g., the Internet, WIFI, ultra-wideband, or Bluetooth). The capacity of the wireless network is limited and may vary over time based on factors such as the number of other wireless devices nearby, the distance between the EAMS and the HMD 110, the arrangement of other equipment, walls, and people in the operating room, etc. EAMS data may be delayed due to network conditions, causing the location of the catheter or medical tool displayed by the HMD 110 to fall behind the actual location of the catheter or medical tool in the patient, which in turn may make it more difficult for the doctor to steer the catheter to the intended target.

[0103] There is generally a trade-off between the level of detail or fidelity in a computer-generated image and the amount of data that needs to be sent to reproduce the image in the HMD 110. In some embodiments, the HMD 110 can measure network reception strength, signal-to-noise level, latency, or throughput, and feed that data back to the EAMS. Based on that feedback, the EAMS can adjust the level of detail in the data sent to the HMD 110, forming a closed-loop control system. For example, when the throughput of the network is less than the rate at which the EAMS is sending data, the EAMS can throttle the data being sent in a way that makes it more likely to arrive at the HMD 110 in a timely manner.

[0104] In some embodiments, the EAMS computing device 102 sends (e.g., via the data computing device 104) the shape of the patient's anatomical structure as a 3D mesh of triangles or as a collection of 3D voxels to the HMD 110. EAMS can reduce the number of triangles, the accuracy of the vertex positions of the triangles, the resolution of the 3D texture, or any combination thereof. These actions result in a less detailed image of the anatomical structure displayed by the HMD 110. In some embodiments, EAMS further reduces the details in areas of the anatomical structure that are farther away from the viewing angle of the HMD 110 (referred to as foveated rendering by those skilled in the art) or farther away from the location of a medical tool (such as the distal tip of a catheter).

[0105] In some embodiments, when the catheter moves slowly, EAMS sends data to HMD 110 at a higher level of detail because transmission time is not a limiting factor. Conversely, when the catheter moves quickly, EAMS sends data to HMD 110 at a lower level of detail but at a faster transmission rate to ensure that HMD 110 is updated in a timely manner to reflect the catheter movement.

[0106] In some embodiments, when the network speed decreases, EAMS reduces the level of detail in the position and shape of the catheter or medical tool. For example, EAMS reduces the precision in the position or reduces the number of control points in the spline. EAMS allocates more details or bandwidth to catheters that move quickly (e.g., above a rate threshold of change or relative to the movement of other catheters) or have distal tips that are closer to the HMD gaze point. EAMS can determine a greater image update rate for catheters that move more quickly (e.g., through user-manipulated movement or through system control). EAMS can determine a greater image update rate for catheters that are closer to the HMD gaze point.

[0107] XIII. Additional Configuration

[0108] The disclosed embodiments are not mutually exclusive. The disclosed embodiments may be combined into embodiments to provide multiple options to a user during a medical procedure. For example, a user may begin specifying 3D spatial relationships according to a handheld controller embodiment, then put down the handheld controller and continue to refine the 3D spatial relationships according to a hands-free embodiment.

[0109] The foregoing description of the embodiments of the present disclosure has been presented for illustrative purposes; it is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0110] Some parts of this description describe embodiments of the present disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are usually used by technicians in the field of data processing to efficiently convey the essence of their work to other technicians in the field. Although these operations are described functionally, computationally, or logically, these operations are understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it is sometimes proven to be convenient to refer to the arrangement of the operation as a module. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.

[0111] Any steps, operations, or processes described herein may be performed or implemented using one or more hardware modules or software modules, either alone or in combination with other devices. In some embodiments, a software module is implemented using a computer program product, which includes a non-transient computer-readable storage medium, which includes a computer program code, which may be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0112] Embodiments may also relate to products produced by the computing processes described herein. Such products may include information produced by the computing processes, wherein the information is stored on a non-transitory, tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.

[0113] Finally, the language used in the specification is selected primarily for readability and instructional purposes, and the language cannot be selected to describe or constrain the subject matter of the present invention. Therefore, the scope of the present invention is not intended to be limited by this detailed description, but by any claims issued on the application based thereon. Therefore, the disclosure of the embodiments herein is intended to illustrate rather than limit the scope of the present invention, and the scope of the present invention is set forth in the appended claims.

Claims

1. A system comprising: a head mounted display (HMD) worn by a user; Display monitor; as well as A processing system configured to: providing image data for display by the HMD based on data provided to the display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; Determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; as well as User input information is provided to the display monitor to update the user interface based on the user input and the position of the cursor.

2. The system of claim 1 , wherein the processing system comprises a frame grabber configured to: The data is received that is provided to the display monitor for the user interface displayed by the display monitor.

3. The system of claim 1, wherein the processing system comprises: a first computing device; a second computing device configured to provide the image data for display by the HMD and to receive the user input from the HMD; as well as A signal generator is configured to send information associated with the user input to the first computing device, wherein the first computing device generates the user input information using the information associated with the user input.

4. The system of claim 3, wherein the signal generator sends the information associated with the user input as a USB signal, and wherein the first computing device generates the user input information without using input from a mouse or keyboard.

5. The system of claim 3, wherein the second computing device receives the gaze direction from the user of the HMD, and wherein the first computing device provides the position of the cursor to the display monitor.

6. The system of claim 1, wherein the HMD is further configured to: displaying augmented reality graphics; and In response to determining that the user is interacting with the user interface, the display of the augmented reality graphics is updated to track the position of the cursor.

7. The system of claim 1, wherein the processing system is further configured to: The action performed by the user is determined to be a selection of a user control among a plurality of user controls displayed in the user interface.

8. The system of claim 1, wherein the processing system is further configured to: Provides registration information to the 3D anatomical information source.

9. A method comprising: providing image data for display by a head mounted display (HMD) worn by a user based on data provided to a display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; Determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; as well as User input information is provided to the display monitor to update the user interface based on the user input and the position of the cursor.

10. The method according to claim 9, further comprising: The data is received that is provided to the display monitor for the user interface displayed by the display monitor.

11. The method according to claim 9, further comprising: Information associated with the user input is sent by a signal generator from the second computing device to the first computing device, wherein the first computing device generates the user input information using the information associated with the user input.

12. The method of claim 11, wherein the signal generator sends the information associated with the user input as a USB signal, and wherein the first computing device generates the user input information without using input from a mouse or keyboard.

13. The method of claim 11, wherein the second computing device receives the gaze direction from the user of the HMD, and wherein the first computing device provides the position of the cursor to the display monitor.

14. The method according to claim 9, further comprising: displaying augmented reality graphics by the HMD; as well as In response to determining that the user is interacting with the user interface, updating the display of the augmented reality graphics to track the position of the cursor.

15. The method according to claim 9, further comprising: The action performed by the user is determined to be a selection of a user control among a plurality of user controls displayed in the user interface.

16. The method according to claim 9, further comprising: Provides registration information to the 3D anatomical information source.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: providing image data for display by a head mounted display (HMD) worn by a user based on data provided to a display monitor for a user interface displayed by the display monitor; receiving a gaze direction of the user from the HMD; Determining a position of a cursor displayed on the user interface based on the gaze direction of the user; providing the position of the cursor to the display monitor to update a display of the cursor on the display monitor; receiving user input from the HMD in response to an action performed by the user; as well as User input information is provided to the display monitor to update the user interface based on the user input and the position of the cursor.

18. The non-transitory computer readable storage medium of claim 17, storing further instructions that, when executed by the one or more processors, cause the one or more processors to: The data is received that is provided to the display monitor for the user interface displayed by the display monitor.

19. The non-transitory computer-readable storage medium of claim 17, the non-transitory computer-readable storage medium storing further instructions that, when executed by the one or more processors, cause the one or more processors to: Information associated with the user input is sent by a signal generator from the second computing device to the first computing device, wherein the first computing device generates the user input information using the information associated with the user input.

20. The non-transitory computer-readable storage medium of claim 17, storing further instructions that, when executed by the one or more processors, cause the one or more processors to: displaying augmented reality graphics by the HMD; and In response to determining that the user is interacting with the user interface, the display of the augmented reality graphics is updated to track the position of the cursor.

Images