System and method for aligning the movement direction of an interventional device in an image and the control direction of a command entered by a user - Patents.com

Abstract

A system and method for aligning a movement of an interventional device in an image with a control command input by a user to visualize the progression of a robotically guided interventional device in an anatomical structure is provided, the method includes the steps of receiving a current image of the interventional device in a current position, receiving a control input from an input device on a control console for controlling the movement of the interventional device, determining a control direction of the input device, estimating a movement direction of the interventional device in the current image based on the control input, estimating a discrepancy between the movement direction and the control direction, and adjusting an orientation of the current image relative to a display or an orientation of the input device relative to the control console to align the movement direction of the interventional device with the control direction of the input device of the control console.

Description

[Technical field]

[0001] The present invention relates to a system and method for aligning the direction of movement of an interventional device in an image and the control direction of commands entered by a user. [Background technology]

[0002] Robotic systems used to perform interventional procedures, such as endovascular procedures, are operated by a user using a control console of a robotic controller to steer a bendable interventional device, such as a catheter or guidewire. The movement and behavior of the interventional device in the subject (patient) depends on control inputs provided by the user through the control console, as well as the shape, properties and location of the interventional device relative to the subject's anatomy. The control inputs may be generated using a control interface of the control console, such as a joystick, where directional movement of the control interface causes corresponding directional movement of the interventional device displayed on a screen.

[0003] During an interventional procedure, a user typically navigates an interventional device through complex anatomical structures (e.g., blood vessels) with high curvatures, branches, and tortuous paths while viewing the progression in a displayed image, which can be difficult and time consuming. Improper navigation can result in multiple failed attempts at performing an interventional procedure, resulting in damage to the anatomical structures (e.g., bleeding), increased procedure time, increased exposure of the subject to imaging radiation, and / or the need to change the interventional device. Summary of the Invention [Problem to be solved by the invention]

[0004] When navigating an interventional instrument under real-time imaging guidance (e.g., fluoroscopy), a user's hand-eye coordination is a key concern. Facilitating proper hand-eye coordination is particularly difficult when the interventional device is remotely operated from a robotic controller, e.g., when the control console is remote from the subject, e.g., due to radiation safety or other concerns. However, hand-eye coordination is difficult when the control direction entered into the control console does not match the movement direction of the interventional device in the displayed image. For example, the interventional device may be shown to move to the left in the displayed image while the user enters a control input by pushing a joystick upward on the control console. Such misalignments challenge the user's hand-eye coordination when navigating an interventional device when the steering input is not intuitive relative to the movement direction of the interventional device. [Means for solving the problem]

[0005] According to a representative embodiment, a system is provided for aligning a movement of an interventional device in an image on a display with a control command input by a user to visualize a progression of an interventional device guided by a robot and configured for insertion into a target anatomy. The system includes a display configured to display an image of the interventional device in the target anatomy, a control console including an input device operable by a user to control the movement of the interventional device via the robot, and at least one processor coupled to the display and the control console. The at least one processor is configured to receive a current image of the interventional device in the anatomy displayed on the display, the current image indicating a current position of the interventional device, receive a control input from the control console for controlling the movement of the interventional device from the current position, determine a control direction of the input device relative to the control console based on the control input, estimate a movement direction of the interventional device in the current image on the display based on the control input, estimate a mismatch between the movement direction of the interventional device and the control direction of the input device of the control console, and adjust an orientation of the current image relative to the display or an orientation of the input device relative to the control console to align the movement direction of the interventional device on the display with the control direction of the input device of the control console.

[0006] According to another representative embodiment, a system for displaying and controlling progression of an interventional device configured for insertion into a target anatomical structure is provided, the system comprising: at least one processor coupled to (i) a display and (ii) a user interface for providing control inputs for controlling movement of the interventional device;

[0007] Retrieving a determinate coordinate system associated with a user interface stored in said memory;

[0008] receiving image data of a current image of an interventional device in the anatomical structure displayed or to be displayed on a display, the current image indicating a current position of the interventional device; receiving a control input from a user interface for controlling a movement of the interventional device from a current position, the control input representing a control direction in said predefined coordinate system of the user interface; estimating, from at least the current image data, a direction of movement of the interventional device based on the control input; Estimating a discrepancy between a movement direction of the interventional device and a control direction; determining a change in orientation of the current image displayed or to be displayed or a change in orientation of a coordinate system of the user interface in order to align a movement direction of the interventional device in the current image and a control direction of the user interface; Implementing orientation changes that may occur relative to the display or user interface; It is configured as follows.

[0009] The invention and specification further includes the subject matter of any and all dependent claims 2 to 15.

[0010] According to another representative embodiment, a method is provided for aligning movement of an interventional device in an image on a display with control commands input by a user using a control console to visualize a progression of an interventional device guided by a robot and configured for insertion into a target anatomy, the method comprising the steps of receiving a current image of the interventional device in the target anatomy, the current image being displayed on the display and showing a current position of the interventional device, receiving a control input from the control console in response to an actuation by a user of an input device for controlling movement of the interventional device from the current position via the robot, determining a control direction of the input device relative to the control console based on the control input, estimating a movement direction of the interventional device in the current image on the display based on the control input, estimating a discrepancy between the movement direction of the interventional device and the control direction of the input device of the control console, and adjusting an orientation of the current image relative to the display or an orientation of the input device relative to the control console to align the movement direction of the interventional device on the display with the control direction of the input device of the control console.

[0011] According to another representative embodiment, a method for displaying and controlling progression of an interventional device configured for insertion into a target anatomical structure is provided, the method comprising: receiving current image data of an interventional device in a target anatomy, the current image being or being displayed on a display and indicating a current position of the interventional device; receiving a control input from a user interface for controlling a movement of the interventional device from a current position, e.g. in response to a user's actuation of an input device, the control input representing a control direction in an aligned predefined coordinate system of the user interface; estimating, from at least the current image data, a direction of movement of the interventional device based on the control input; - estimating a discrepancy between a movement direction and a control direction of the interventional device; - determining a change in orientation of the current image or a change in orientation of a coordinate system of the user interface in order to align a movement direction of the interventional device in the current image and a control direction of the user interface; implementing an orientation change, which may be made relative to a display or user interface; has.

[0012] According to another representative embodiment, a non-transitory computer-readable medium is provided that stores instructions for aligning movement of an interventional device in an image on a display with control commands input by a user using a control console to visualize a progression of an interventional device configured for insertion into a target anatomy guided by a robot, wherein the instructions, when executed by at least one processor, cause the at least one processor to receive a current image of the interventional device in the target anatomy, the current image being displayed on the display indicating a current position of the interventional device, receive a control input from the control console in response to an actuation by a user of an input device to control movement of the interventional device from the current position via the robot, determine a control direction of the input device relative to the control console based on the control input, estimate a movement direction of the interventional device in the current image on the display based on the control input, estimate a mismatch between the movement direction of the interventional device and the control direction of the input device of the control console, and adjust an orientation of the current image relative to the display or an orientation of the input device relative to the control console to align the movement direction of the interventional device on the display with the control direction of the input device of the control console.

[0013] According to another representative embodiment, a non-transitory computer readable medium storing instructions for displaying and controlling advancement of an interventional device configured for insertion into a target anatomical structure, the instructions, when executed by at least one processor adapted to be coupled to (i) a display and (ii) a user interface for providing control inputs for controlling movement of the interventional device, causes the at least one processor to: Read the default coordinate system associated with the user interface; receiving current image data of an interventional device within an anatomical structure that is or will be displayed on a display, the current image indicating a current position of the interventional device; receiving a control input from a user interface for controlling a movement of the interventional device from a current position, the control input representing a control direction in said predefined coordinate system of the user interface; estimating a direction of movement of the interventional device based on a control input from at least the current image data; Estimating a discrepancy between a movement direction of the interventional device and a control direction; determining a change in orientation of the current image or a change in orientation of a coordinate system of the user interface to align a direction of movement of the interventional device in the current image and a control direction of the user interface; It implements orientation changes that may occur relative to a display or user interface.

[0014] The illustrative embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements. [Brief description of the drawings]

[0015] [Figure 1A] 1 illustrates an exemplary control console for controlling movement of an interventional device in an exemplary display image having inconsistent control and movement directions. [Figure 1B] 1 illustrates an exemplary control console and an exemplary displayed image of an interventional device with matching control and movement directions with which the image is oriented, according to a representative embodiment. [Figure 1C]1 illustrates an exemplary control console and an exemplary display image of an interventional device with matching control and movement directions to which the orientation of an input device on the control console is adjusted, according to a representative embodiment. [Diagram 2] FIG. 1 is a simplified block diagram of a system for aligning movement of an interventional device in a displayed image with directional control commands input by a user through a control console to visualize progression of the interventional device within a target anatomical structure, according to a representative embodiment. [Diagram 3] FIG. 1 is a flow diagram illustrating a method for aligning movement of an interventional device in a displayed image with directional control commands entered by a user through a control console to visualize the progression of the interventional device within a target anatomical structure, according to a representative embodiment. [Figure 4] 1 shows exemplary display images in which the movement direction of the interventional device in each image is estimated by inferring the shape and position of the surrounding anatomical tissue and the interventional device from the current image and recent past images of the subject, in accordance with a representative embodiment. [Diagram 5] 1 illustrates an exemplary display image in which a movement direction of an interventional device in a current image is estimated by applying a first neural network model to obtain a future motion vector for predicting a corresponding next image indicating a future direction of movement of the interventional device, according to a representative embodiment. [Figure 6A] 1 illustrates an exemplary control console for controlling directional movement of an interventional device and an exemplary display image of the interventional device as viewed straight ahead by a user, according to a representative embodiment. [Figure 6B] 1 illustrates an exemplary control console for controlling directional movement of an interventional device, according to a representative embodiment, and an exemplary display image of the interventional device as seen by a user at an offset angle with a matching directional orientation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted to avoid obscuring the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the scope of those skilled in the art may be used in accordance with the representative embodiments, within the scope of the present teachings. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. Defined terms are given the technical and scientific meaning of the defined terms as commonly understood and accepted in the art of the present teachings.

[0017] Terms such as first, second, third, etc. may be used herein to describe various elements or components, but it should be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another element or component. Thus, a first element or component discussed below can be referred to as a second element or component without departing from the teachings of the inventive concept.

[0018] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in this specification and the appended claims, the singular forms of the terms "a", "an" and "the" are intended to include both the singular and the plural, unless the context clearly dictates otherwise. In addition, the terms "have", "comprises", and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] Unless otherwise stated, when an element or component is said to be "connected," "coupled," or "adjacent" to another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or there may be intervening elements or components. That is, these and similar terms encompass the cases where one or more intermediate elements or components may be used to connect the two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this only encompasses the cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0020] Thus, the present disclosure is intended to derive one or more of the advantages as specifically described below through one or more of its various aspects, embodiments, and / or specific features or subcomponents. For purposes of explanation and not limitation, exemplary embodiments disclosing specific details are described to provide a thorough understanding of embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are within the scope of the appended claims. Furthermore, descriptions of known devices and methods may be omitted so as not to obscure the description of the exemplary embodiments. Such methods and devices are within the scope of the present disclosure.

[0021] In general, various embodiments described herein provide systems and methods that allow a user of a robotic system (interventionalist) to align the control direction of an input device operated on the control console of a robotic controller with the movement direction of an interventional device guided by the robot and visualized in a surgical display during an interventional procedure (e.g., catheter insertion). To achieve this alignment, the image on the display is reoriented to match the orientation of the command entered through the control console, or the directionality of the command entered through the control console is reoriented to match the visualization. For example, self-supervised neural network learning may be used to increase stability and improve the user experience when rotating the displayed image by learning to set visualization parameters based on upcoming events in the procedure. Aligning the control direction associated with the control command with the movement direction of the interventional device provides visualization results that correspond to a more natural hand-eye coordination of the user, which improves the manipulation of the interventional device inside the target anatomy (e.g., vasculature) under fluoroscopic guidance.

[0022] The optimal view orientation may be estimated, for example, by combining robotics data (e.g., kinematics, encoder data, controller inputs) from a robot guiding the interventional device with past image acquisitions showing the interventional device and surrounding tissue, current image acquisitions showing the interventional device and surrounding tissue, and a prediction of the future shape and position of the interventional device. Incorporating the future shape and position improves visualization stability by allowing the data to be filtered in time to provide a smooth transition between displayed images.

[0023] FIGURE 1A illustrates an exemplary control console for controlling movement of an interventional device in an exemplary display image with inconsistent control and movement directions, FIGURE 1B illustrates an exemplary control console and an exemplary display image of an interventional device with matching control and movement directions where the image orientation is adjusted, according to a representative embodiment, and FIGURE 1C illustrates an exemplary control console and an exemplary display image of an interventional device with matching control and movement directions where the orientation of an input device on the control console is adjusted, according to a representative embodiment.

[0024] 1A, a display 124 shows an image 125 of an interventional device 146 inserted into an anatomical structure 155 of a subject 150, such as, for example, a blood vessel or artery. The interventional device 146 is robotically guided under the control of a robot controller, such as robot 144 and robot controller 142, described below with reference to FIG. 2. The interventional device 146 can be any compatible (non-rigid) medical instrument that is robotically controllable, such as, for example, a catheter, guidewire, stent, balloon, sheath, endoscope, camera, or surgical tool.

[0025] The control console 143 is configured to interface with the robotic controller 142 to control the movement of the interventional device 146. The control console 143 includes an input device 145 operable by a user to control the directional movement of the interventional device 146 by applying an input control to the input device 145. The input device 145 may be any compatible interface mechanism operable to indicate the direction of movement of the interventional device 146, such as, for example, a joystick, thumb stick, or directional pad. The input device 145 may also be configured to control the speed of movement of the interventional device 146.

[0026] The display 124 may be continuously updated as additional images are acquired and / or as the user changes control inputs. The images 125 may be, for example, live fluoroscopic images, but may incorporate any other type of image acquired in real time or near real time, such as ultrasound images, X-ray images, computed tomography (CT) images, cone beam CT images, magnetic resonance (MR) images, and positron emission tomography (PET) images, without departing from the scope of the present teachings.

[0027] It should be appreciated that the control console 143 may be any type of console capable of interfacing with a robot controller, including a console that may be specifically designed for interaction with a particular robot controller or an off-the-shelf console that may be programmed for interaction with a particular robot controller, as would be apparent to one skilled in the art. For example, the control console 143 in the illustrated embodiment is a specially programmed handheld Xbox® Wireless Controller available from Microsoft® Corporation, and the input device 145 is the left stick on the Xbox® Wireless Controller. For example, the directional orientation of the input device 145 may be changed by reprogramming the application programming interface (API) of the input device 145. It should be further appreciated that the input device 145 may be implemented as a single mechanism (e.g., a mechanism that controls both direction and speed) or as multiple mechanisms that operate in concert with each other (e.g., one controls direction and one controls speed) without departing from the scope of the present teachings.

[0028] Control of input device 145 D C is shown as an arrow to the control console 143. C is the direction of the control input given by the user to the input device 145. For the sake of explanation, the control direction D C is the virtual control axis x C , y C In the illustrated example, the control direction DC is directly above (+y C 1. In the embodiment shown in FIG. 1, the input device 145 is shown pointing in the upright (direction) direction, indicating that the input device 145 is being pushed upward by a user to steer the interventional device 146 straight ahead (forward).

[0029] Direction of movement D of the intervention device 146 M is shown as an arrow at the distal end of the interventional device 146 relative to the display 124. M responds to the operation (control input) of the input device 145. For the sake of explanation, the movement direction D M is the virtual moving axis x M , y M In the illustrated example, the direction of movement D M is the left (-x M 125), which means that in response to the input device 145 being moved upward relative to the control console 143, the distal end of the interventional device 146 moves straight ahead (forward) by moving left relative to the display 124. This may cause some confusion to a user who may be intuitively induced to move the input device 145 to the left, which would cause the interventional device 146 to move forward in the current orientation of the image 125. However, this control input actually causes the interventional device 146 to rotate left and move downward relative to the display 124.

[0030] In contrast, FIG. 1B illustrates a cross-sectional view of a movement direction D of an interventional device 146 according to a representative embodiment. M is the control direction D of the input device 145 C 1 shows the control console 143 and the display 124 after the image 125 has been reoriented (e.g., rotated) on the display 124 so as to visually align with the control direction D C is shown pointing straight up, still indicating that the input device 145 is being pushed upward by the user to manipulate the interventional device 146 forward. However, the direction of movement D M is the control direction D C For illustration purposes, the direction of movement DM This reorientation of the display 124 is performed by rotating the axis of translation x M , y M Therefore, the direction of movement D M indicates that the distal end of the interventional device 146 moves forward in the image 125 by moving upward relative to the display 124 in response to the input device 145 being moved upward relative to the control console 143. Thus, a user can intuitively move the input device 145 in the same direction as the movement of the interventional device 146 to intuitively control the interventional device 146 to move forward.

[0031] In an alternative embodiment, referring to FIG. 1C, the orientation of the image 125 and the direction of movement D of the interventional device 146 are M remains unchanged, while the control input of the input device 145 is in the control direction D C is the direction of movement D M 1A remains the same, which means that the movement direction of the interventional device 146 is to the left when it is moved forward. On the other hand, the orientation of the input device 145 is changed relative to the control device 143, resulting in a control direction D C Moving the input device 145 to the left so that it points left causes the intervention device 146 to move forward in the image 125 .

[0032] That is, FIG. 1C shows the control direction D of the input device 145. C is the direction of movement D of the interventional device 146 in the image 125 M 1 shows the control console 143 and display 124 after the input device 145 has been reoriented (e.g., rotated) relative to the control console 143 so as to visually align with the direction of movement D M is shown as an arrow pointing to the left, indicating that the interventional device 146 is still moving to the left in the image 125 when it is controlled to move forward. Cis also shown pointing to the left, indicating that the input device 145 has been reoriented such that the user is pushing the input device 145 to the left to manipulate the intervention device 146 forward. C This reorientation of the control axis x is rotated counterclockwise relative to the control console 143. C , y C Thus, the user can intuitively move the input device 145 in the same direction as the movement of the interventional device 146 to control the interventional device 146 to move forward.

[0033] In various embodiments, the interventional device 146 can be a coaxial device that includes an inner device and a surrounding outer device. For example, the interventional device 146 can include a guidewire inserted through a catheter, each of which is separately controllable. In this case, the user can control the direction D. C and the moving direction D M A user may select one of the inner and outer devices to control the determination of the orientation of the image 125 on the display 124 for alignment with the image 125. That is, the user may select the most distal of the inner and outer devices as a reference for determining the alignment, the most proximal of the inner and outer devices as a reference for determining the alignment, or an average orientation of the inner and outer devices for determining the alignment.

[0034] Other reference metrics of the interventional device 146 (whether coaxial or not) for estimating a desired orientation alignment include considering the shape and / or orientation of the interventional device 146 for a given section of N millimeters and / or pixels, a section of the interventional device 146 that is actively steerable, or a distal section of the interventional device that is, for example, straight. If the interventional device 146 is an articulated device, the orientation alignment may be determined based, for example, on the most distal segment of the interventional device, the most proximal segment of the interventional device, an average of the N distal segments, or an average of the N proximal segments, where N is a positive integer greater than 1.

[0035] FIG. 2 is a simplified block diagram of a system for aligning movement of an interventional device in a displayed image with directional control commands input by a user through a control console to visualize the progression of the interventional device within a target anatomical structure, according to a representative embodiment.

[0036] 2, the system 100 includes a workstation 105 for implementing and / or managing the processes described herein with respect to aligning the movement of an interventional device 146 in an image 125 on a display 124 with control commands input by a user to visualize the progression of the interventional device 146 in an anatomical structure 155 of a subject (patient) 150. The workstation 105 includes one or more processors represented by a processor 120, one or more memories represented by a memory 130, a user interface 122, and a display 124. The processor 120 interfaces with a robotic system 140 via a control module 132, the robotic system 140 including a robot controller 142, a control console 143, and a robot 144. The robot controller 142 is configured to control the movement of the robot 144 in response to user control inputs received by manipulation of an input device 145 of the control console 143. The robot 144 is attached to or integrated with the interventional device 146. The robot 144 may include segments, joints, servo motors, and other control features operable to move and position the interventional device 146 with multiple degrees of freedom (DOF) in response to control signals received from the robot controller 142. In the illustrated embodiment, the robot controller 142 is shown separate from the processor 120 in the workstation 105 for illustrative purposes. However, it will be understood that all or a portion of the functionality of the robot controller 142 may be incorporated into the processor 120, or vice versa, without departing from the scope of the present teachings.

[0037] A user interfaces with the robot controller 142 using a control console 143. The control console 143 may be a handheld control console such as the specially programmed Xbox Wireless Controller available from Microsoft Corporation as mentioned above, although any type of compatible control console may be incorporated without departing from the scope of the present teachings. The control console 143 may communicate with the robot controller 142 via a wireless connection, as indicated by the dashed lines, such as Bluetooth (IEEE 802.15.1), ZigBee (IEEE 802.15.4), or WiFi (IEEE 802.11), for example, directly or via a local or wide area network. Alternatively, the control console 143 may communicate with the robot controller 142 via a wired connection, such as, for example, a transmission line, cable, coaxial cable, or fiber optic cable.

[0038] Processor 120 also interfaces with imaging device 160 via imaging module 131. Imaging device 160 can be any of a variety of types of medical imaging devices / modalities including, for example, a fixed or mobile C-arm fluoroscopy system, an X-ray imaging device, a CT scanning device, an MR imaging device, a PET scanning device, or an ultrasound imaging device. Imaging device 160 may include a single or multiple imaging modalities.

[0039] Memory 130 stores instructions executable by processor 120. When executed, the instructions cause processor 120 to perform one or more processes for aligning movement of interventional device 146 in image 125 with control commands entered by a user through control console 143 to intuitively visualize the progression of interventional device 146 within anatomical structure 155. For purposes of illustration, memory 130 is shown to include software modules, each of which includes instructions corresponding to an associated capability of system 100, as described below.

[0040] Processor 120 represents one or more processing devices, which may be implemented using any combination of hardware, software, firmware, hardwired logic circuitry, or combinations thereof, by a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a general purpose computer, a central processing unit, a graphic processing unit, a computer processor, a microprocessor, a microcontroller, a state machine, a programmable logic device, or a combination thereof. Any processing unit or processor herein may include multiple processors, parallel processors, or both. Multiple processors may be included in or combined with a single device or multiple devices. The term "processor" as used herein encompasses electronic components capable of executing programs or machine-executable instructions. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems, such as a cloud-based or other multi-site application. A program has software instructions executed by one or more processors, which may be within the same computing device or distributed across multiple computing devices.

[0041] Memory 130 may include main memory and / or static memory, and such memories may communicate with each other and with processor 120 via one or more buses. Memory 130 may be implemented by any number, type, and combination of, for example, random access memory (RAM) and read-only memory (ROM) and may store various types of information, such as software algorithms, artificial intelligence (AI) machine learning models, and computer programs, all of which are executable by processor 120. The various types of ROM and RAM may include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, electrically field programmable gate array read-only memory (EPROM), electrically erasable and field programmable gate array read-only memory (EEPROM), registers, hard disks, removable disks, tape, compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), floppy disks, Blu-ray disks, universal serial bus (USB) drives, or any other form of storage media known in the art. Memory 130 is a tangible storage medium that stores data and executable software instructions and is non-transient for the time that the software instructions are stored. As used herein, the term "non-transient" should be interpreted as a characteristic of a state that persists over a period of time, rather than as a permanent characteristic of a state. The term "non-transient" specifically negates ephemeral characteristics such as those of a carrier wave or signal, or other form that exists only temporarily at any place at any time. Memory 130 may store software instructions and / or computer readable code that enable the performance of various functions. Memory 130 may be secure and / or encrypted, or non-secure and / or unencrypted.

[0042] The system 100 may also include a database 112 for storing information that may be used by the various software modules in the memory 130. For example, the database 112 may include image data from previously acquired images of the subject 150 and / or other similarly positioned subjects having the same or similar interventional procedure as the subject 150, along with control input data indicative of control inputs to the robot corresponding to the previously acquired images. The stored image data and corresponding control input data may be used, for example, to train an AI machine learning model, such as a neural network model, as discussed below. The database 112 may be implemented, for example, by any number, type, and combination of RAM and ROM. The various types of ROM and RAM may include any number, type, and combination of computer readable storage media, such as disk drives, flash memory, EPROM, EEPROM, registers, hard disks, removable disks, tapes, CD-ROMs, DVDs, floppy disks, Blu-ray disks, USB drives, or any other form of storage media known in the art. The database 112 comprises a tangible storage medium for storing data and executable software instructions and is non-transient during the time the data and software instructions are stored. The database 112 may be secure and / or encrypted, or may be non-secure and / or unencrypted. For purposes of illustration, the database 112 is shown as a separate storage medium, but it will be understood that it may be combined with and / or included in the memory 130 without departing from the scope of the present teachings.

[0043] Processor 120 may include or have access to an AI engine, which may be implemented as software that provides artificial intelligence (e.g., neural network models) and applies machine learning as described herein. The AI ​​engine may reside in addition to processor 120 or in any of a variety of components other than processor 120, such as, for example, memory 130, an external server, and / or the cloud. If the AI ​​engine is implemented in the cloud, such as, for example, a data center, the AI ​​engine may be connected to processor 120 via the Internet using one or more wired and / or wireless connections.

[0044] The user interface 122 is configured to provide information and data output by the processor 120, memory 130, and / or robot controller 142 to a user and / or to receive information and data input by a user. That is, the user interface 122 allows a user to input data and control or manipulate aspects of the processes described herein, and also allows the processor 120 to indicate the effect of the user's input. All or part of the user interface 122 may be implemented by a graphical user interface (GUI), such as a GUI 128 visible on a screen 126 described below. The user interface 122 may include one or more interface devices, such as, for example, a mouse, a keyboard, a trackball, a joystick, a microphone, a video camera, a touchpad, a touch screen, voice or gesture recognition captured by a microphone or video camera, etc.

[0045] Display 124 may be, for example, a monitor such as a computer monitor, a television, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT) display, or an electronic whiteboard. Display 124 includes a screen 126 for viewing an image (e.g., image 125) of object 150 along with various features described herein to assist the user in reading the image accurately and efficiently, and a GUI 128 for allowing the user to interact with the displayed images and features. The user may personalize various features of GUI 128, described below, for example, by creating specific alerts and reminders.

[0046] With reference to the memory 130, the various modules store sets of data and instructions executable by the processor 120 to align the displayed movement of the interventional device 146 in the image 125 with the direction of control commands entered by the user through the control console 143 to intuitively visualize the progression of the interventional device 146 within the anatomical structure 155.

[0047] The imaging module 131 is configured to receive and process images of the anatomical structures 155 in the subject 150 and the interventional device 146, including a series of current images 125 viewed by a user during an interventional procedure. Each image 125 may be received in real time from the imaging device 160 during a concurrent current imaging session of the subject 150. Alternatively, the images 125 may be images of the subject 150 acquired previously in the current imaging session, for example when imaging was paused to reduce exposure of the subject 150 to radiation. Similarly, the images 125 may be retrieved from a database 112, which stores images acquired during a previous imaging session(s) (from a single or multiple imaging modalities) or earlier than the current imaging session. The current images 125 are displayed on a screen 126 to enable analysis by the user and navigation of the interventional device 146 via the robot 144.

[0048] The control module 132 is configured to receive control inputs from a user via the control console 143 and the robot controller 142 to control the robot 144 to guide the movement of the interventional device 146, and to determine a control direction of the input device 145 relative to the control console 143 based on the control inputs. The control inputs include, for example, steering commands such as articulation, rotation, translation, distance, velocity, and acceleration of the robot 144 to move the interventional device 146 within the anatomical structure 155. The control direction is the direction in which the user manipulates the input device 145 on the control console 143 to control the movement of the interventional device 146. For example, the control direction may be the direction of the control axis x of the control console 143, as shown in FIGS. 1A and 1B . C , y C In this case, the control direction can be determined by the control axis x from the origin to the x, y coordinates. C , y C , and the length of the vector may be determined, for example, by the amount of time the user holds the input device in a desired control direction. The control direction of the input device 145 may be reoriented by the control module 132 with respect to the control console 143, which may be referred to as reorienting the input device. For example, manipulating the input device 145 straight up (up) may be reoriented from moving the interventional device 146 forward to moving the interventional device 146 to the left. In this manner, the control direction D of the input device 145 may be reoriented by the control module 132 with respect to the control console 143, which may be referred to as reorienting the input device. For example, manipulating the input device 145 straight up (up) may be reoriented from moving the interventional device 146 forward to moving the interventional device 146 to the left. C is the movement direction D of the interventional device shown on the display 124 M can be aligned with

[0049] In one embodiment, the control input may be initially untriggered, in that the user enters the control input without it being executed by the robot controller 142. In this manner, the processor 120 can predict the effect of the control input on the trajectory of the interventional device 146 before the robot controller 142 controls the robot 144 to actually move the interventional device 146 in response to the control input. This allows the user to determine whether an untriggered control input is appropriate before action is taken to implement the control input.

[0050] The control / display alignment module 133 is configured to align the movement of the interventional device 146 in the current image 125 on the display 124 with the control input provided by the control module 132 to allow the user to visualize the progress of the interventional device 146 while intuitively inputting the control input. In one embodiment, as described in more detail below with reference to FIG. 3, the control / display alignment module 133 estimates a movement direction of the interventional device 146 in the current image 125 based on the control input. The estimated movement direction is the most likely direction in which the interventional device 146 moves in response to a control input by a user by manipulating the input device 145 on the control console 143. In general, the estimated movement direction is determined from the control input provided by the control module 132 indicating the control direction of the input device 145 and the image data of the current image 125 provided by the imaging module 131. For example, the movement direction may be determined based on a movement axis x of the current image 125 as shown in FIGS. 1A and 1B. M , y M In this case, the direction of movement is from the origin along the movement axis x M ,y M may be represented as a vector extending along x, y coordinates.

[0051] The control / display alignment module 133 estimates a mismatch between the movement direction of the interventional device 146 and the control direction applied to the input device 145 of the control console 143. The control / display alignment module 133 then aligns the movement direction of the interventional device 146 on the display 124 with the control direction of the input device 145 on the control console 143 by compensating for the estimated mismatch, as described below. The movement direction and the control direction can be aligned by adjusting the orientation of the current image 125 on the display 124 such that the movement direction of the interventional device 146 aligns with the control direction of the input device 145, or by adjusting the orientation of the input device 145 relative to the control console 143 such that the control direction aligns with the movement direction of the interventional device 146 in the current image 125 on the display 124. The control / display alignment module 133 may include a neural network model for estimating the movement direction of the interventional device and / or for estimating the mismatch between the movement direction of the interventional device and the control direction of the control input provided by the input device 145, as described below.

[0052] The previous image module 135 is configured to receive previous image data from the previous images, including previous images of the object 150, for example, from the imaging device 160 and / or the database 112. The previous images include images of the object 150 previously acquired in the current imaging session and / or images of the object 150 acquired during a previous imaging session with the same type of interventional procedure. The images may have been acquired from a single or multiple imaging modalities. In an embodiment, the previous image module 135 may also receive previous images of other similarly positioned objects having the same or similar interventional procedure as the object 150 having the same type of interventional procedure. The previous image data of the object 150 and / or other objects may include control inputs corresponding to the previous images, indicating commands entered by a user for positioning when the previous images were acquired. The previous images of the object 150 and / or other objects may be used, for example, to train a neural network model, as discussed below.

[0053] 3 is a flow diagram of a method for aligning movement of an interventional device in a displayed image with directional control commands entered by a user through a control console to visualize progression of the interventional device within a target anatomy, according to a representative embodiment. The method may be implemented by the system 100 described above under control of the processor 120 executing instructions stored as various software modules in the memory 130, for example.

[0054] 3, the method includes, in block S311, receiving a current image (e.g., current image 125) showing an interventional device (e.g., interventional device 146) in an anatomical structure (e.g., anatomy 155), the current image being displayed on a display (e.g., display 124). The current image shows the current position of the interventional device in the anatomical structure. The process described below assumes that the current image is a two-dimensional image, but may be applied to three-dimensional images or three-dimensional fluoroscopic images (four-dimensional images) without departing from the scope of the present teachings. The three-dimensional and four-dimensional image data may be displayed by projecting them onto a two-dimensional display, e.g., as a digitally reconstructed radiograph (DRR), or by displaying two-dimensional slices through the three-dimensional image data, e.g., in an axial or sagittal plane. Using the three-dimensional or four-dimensional image data, the display of the projection images or image slices may be aligned with the robot controller according to the process described herein.

[0055] In block S312, a control input is received from a control console (e.g., control console 143) for controlling movement of the interventional device from a current position. The control input may be initiated by a user at the control console by an action of an input device (e.g., input device 145) and provided to the robot controller for controlling the robot to move the interventional device. The control input may be associated with a corresponding control direction (D C), which is the direction in which an input device on the control console is moved relative to the control console to affect a corresponding movement of the interventional device. For example, if the input device is a joystick, the input device may be pushed upward to cause a forward movement, downward to cause a backward movement, left to cause a left movement, and right to cause a right movement of the interventional device.

[0056] In block S313, a control direction of the input device is determined for the control console based on the control input. The control direction is the direction in which a user manipulates the input device on the control console to control movement of the interventional device. The control direction is the direction in which a user manipulates the input device on the control console to control movement of the interventional device, as discussed above, e.g., the control axis x of the control console 143 as shown in Figures 1A and 1B. C , y C may be determined for

[0057] In S314, the movement direction of the interventional device in the current image on the display (D M ) is estimated based on the control input. The estimated movement direction is the direction in which the interventional device is most likely to move in response to a control input by a user by manipulating an input device on the control console. In general, the estimated movement direction is determined from control data indicating the control direction of the input device and image data from a current image showing the interventional device within the anatomy. For example, the movement direction may be determined based on the movement axis x of image 125 as shown in FIGS. 1A and 1B, as discussed above. M , y M can be estimated for

[0058] In one embodiment, estimating the movement direction of the interventional device includes inferring the shape and position of the surrounding tissue of the anatomical structure and the interventional device from the current image and recent past images of the object. The interventional device and the anatomical structure may be identified in the image using any compatible image recognition technique, such as, for example, edge detection. The image data showing the anatomical structure generally shows a lumen defined by the anatomical structure in which the interventional device is placed, and the walls of the lumen may be identifiable using edge detection. Similarly, the edge detection shows the shape and position of the interventional device within the walls of the lumen, with the longitudinal axis of the interventional device being generally aligned with the longitudinal axis of the anatomical structure. The movement direction of the interventional device may then be estimated based on the shape and position of the surrounding tissue of the anatomical structure and the interventional device within the successive recent past images. For example, the successive recent past images may show the interventional device advancing within the lumen of the anatomical structure along the longitudinal axis. Thus, the estimated movement direction is effectively a projection of the direction established by the shape and position of the interventional device provided by the successive recent past images. Device detection may be accomplished by image segmentation or object detection neural network models, as known in the art, that receive current and / or past images and identify pixels in each image that are occupied by interventional devices and / or anatomical structures.

[0059] In another embodiment, estimating the movement direction of the interventional device includes establishing a current motion vector of the interventional device using the recent past images and corresponding control inputs. The current motion vector represents the direction and magnitude of displacement of the interventional device moving through the anatomical structure as estimated from the recent past images. For example, the current motion vector may be established by locating predetermined points on the interventional device, such as the location of the distal tip or marker, in each of the recent past images and effectively connecting these locations across a predetermined number of recent past images. In one embodiment, the current motion vector may be estimated by applying a first neural network model to the current image and the recent past images. The first neural network model is described in more detail below. Again, the location of the interventional device and the anatomical structure, as well as the predetermined points, may be identified using any compatible shape recognition technique, such as, for example, edge detection.

[0060] A future motion vector is then predicted for the corresponding next image indicating a future direction of the movement of the interventional device based on the current motion vector established from the recent past image. The future motion vector may be predicted by applying a first neural network model to the current motion vector based on the recent past image. In one embodiment, the first neural network model may receive robotics data along with the recent past image to predict the future motion vector. The robotics data may indicate the position and orientation of the robot while controlling the medical instrument in the corresponding image and may include, for example, kinematic data, joint information, encoder information, velocity, acceleration, end effector position, force, torque, and / or range and limit. The first neural network model may be self-supervised, in which case, for example, training the first neural network model may be performed without explicitly labeled data or annotations. According to the first neural network model, the direction of the future motion vector indicates the predicted direction of movement and the length of the future motion vector indicates the number of future frames of the next image required for the movement of the interventional device to be fully realized. In general, the first neural network model compares the current motion vector with a motion vector determined for training images described below, where a similar current motion vector was provided in a similar scenario with respect to the type of anatomical structure and corresponding control inputs used to position the interventional device as shown in the training images. The first neural network model may, for example, include a recursive convolutional layer or a transformer architecture. The movement direction of the interventional device is then estimated based on the predicted future motion vector.

[0061] The first neural network model may be initially trained using training data from training images (e.g., from the previous images module 135) including motion vectors and corresponding control inputs associated with the training images. The training images may be previous images of the same interventional device and corresponding control inputs to the robot for guiding movement of the interventional device through the anatomy of the same subject as shown in the previous images. Additionally or alternatively, the training images may be previous images from other similar interventional procedures of other subjects using the same or similar interventional device and corresponding control inputs to the robot for guiding movement of the interventional device as shown in the previous images.

[0062] The first neural network model is provided in two processes, which may be generally referred to as a training and an inference process. During the training process, appropriate parameters of the first neural network model are learned based on historical data, which includes an optimization process in which the first neural network parameters are changed. The optimization process is iterative, and during each iteration, the first neural network model uses two inputs including a current image and a recent past image from historical cases, and estimates a future motion vector using these two inputs. In one embodiment, the first neural network model may further use current robotics data corresponding to the current image and recent past robotics data corresponding to the recent past image as inputs. The future motion vector may be represented in various forms, such as, for example, key point coordinates at the start and end of the future motion vector. During the inference process, the trained first neural network model predicts the future motion vector. The current and recent past images (and robotics data) are passed forward through the first neural network model during the intervention procedure, and the respective future motion vectors are estimated based thereon.

[0063] More specifically, training the first neural network model may include capturing spatial and temporal context of the previous image. The spatial context may be captured using a convolutional layer consisting of a sliding window or kernel representing a matrix of weights that slides over the input image, performs element-wise multiplication on overlapping portions of the input image, and adds the results to an output feature map. The temporal context may be captured using temporal connections across multiple layers, such as by using a recurrent neural network (RNN), a long short-term memory (LSTM), a transformer, etc.

[0064] Neural network training may be, for example, supervised, self-supervised, or unsupervised. In supervised training, labels are explicitly predefined. In self-supervised training, the current image and control input acquisition may serve as the output (label) for the previous image and corresponding control input acquisition. That is, the subsequent image (and robotics) data may be used as the label for the previous image (and robotics) data points, so that no explicit annotation is required. In unsupervised learning, the image (and robotics) data is clustered such that different clusters indicate different levels of misalignment. For example, one cluster may be associated with image (and robotics) data with a misalignment of 90 degrees, another cluster may be associated with image (and robotics) data with a misalignment of 180 degrees, and so on. Various architectures may be used for unsupervised training, such as, for example, autoencoders and variational autoencoders. During the training phase, the first neural network model learns an appropriate representation from retrospective data, and previous image frames 1, ..., n-1 are used to predict the movement direction of the interventional device at m subsequent time points (image frames n, ..., n+m).

[0065] The previous images and the corresponding control input data may be fed together in the earliest layer of the first neural network model. The previous image data may include, for example, fluoroscopic images or segmentation maps from different devices or anatomical structures in the image. The previous control input data may include measurements from the control console and / or robot controller, the kinematics of the system in terms of joint movements, rotational and translational motions, and velocity and acceleration. Alternatively, the previous control input data may be used in intermediate or latent layers of the first neural network model, acting as a transformation applied to the representation learned from previous image frames 1, ..., n-1. In this case, the control input data may be passed through a series of fully connected layers before merging with the convolutional network. The predicted output in future image frames n, ..., n+m generates different trajectories with different movement directions for different robot transformations or control input data. Another embodiment may use two separate neural networks as the first neural network model, one for the image data and one for the control input data. In this case, the two neural networks share weights or feature maps in some intermediate layers. The training of the first neural network model is performed iteratively, and in each iteration, a batch of corresponding previous images and corresponding previous control input data is fed to the first neural network model. The training is preceded by minimizing a similarity loss, such as binary cross entropy or intensity loss, as will be apparent to those skilled in the art.

[0066] In block S315, a mismatch between the movement direction of the interventional device and the control direction of the input device is estimated. As described above, the control direction is the direction in which the input device on the control console is moved by the user relative to the control console to affect a corresponding movement of the interventional device, and the movement direction is the direction in which the interventional device moves relative to the display in response to the movement of the input device in the control direction. The mismatch refers to the angular difference between the control direction and the movement direction in a common reference space. In Figures 1A and 1B, for example, the mismatch between the movement direction of the interventional device and the control direction of the input device is about 90 degrees, which is about 90 degrees from directly above (e.g., the control axis x C , y C 90 degrees to the left (for example, the x axis) M , y M This means causing a movement of the interventional device to a different position (180 degrees relative to the patient).

[0067] In one embodiment, the mismatch may be estimated by applying a second neural network model to the movement direction of the interventional device and the control direction of the input device. As with the first neural network model, the second neural network model may include supervised, self-supervised, or unsupervised training. In one embodiment, the second neural network model directly outputs a single predicted value representing a rotation that restores the alignment between the movement of the interventional device in the displayed image and the control direction of the directional control command input by the user via the control console. The final layer of the second neural network model may include a function that will normalize the prediction within a finite range (e.g., 0 to 1, or -1 to 1), such as, for example, a softmax function, a sigmoid function, or a hyperbolic tangent (tanh) function. The output may then be rescaled between 0 and 360, which indicates the desired rotation.

[0068] The second neural network model may be initially trained using training data from training images showing the movement direction of the interventional device and the corresponding control direction applied to the input device of the control console in substantially the same manner as described above with respect to the first neural network model. The training images may be previous images of the same interventional device and the corresponding control inputs to the robot for guiding the movement of the interventional device through the anatomy of the same subject as shown in the previous images. Additionally or alternatively, the training images may be previous images from other similar interventional procedures of other subjects using the same or similar interventional device and the corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the previous images. In one embodiment, the second neural network model may also receive robotics data along with the training images to estimate discrepancies between the movement direction of the interventional device and the control direction of the input device. The robotics data may indicate the position and orientation of the robot while controlling the medical instrument in the corresponding training images and may include, for example, kinematic data, joint information, encoder information, velocity, acceleration, end effector position, force, torque, and / or range and limits as described above.

[0069] In block S316, the movement direction of the interventional device on the display and the control direction of the input device are aligned by compensating for the mismatch estimated in block S315. The movement and control directions can be aligned by adjusting the orientation of the current image on the display so that the movement direction of the interventional device aligns with the control direction of the input device, or by adjusting the orientation of the input device relative to the control console so that the control direction aligns with the movement direction of the interventional device in the current image on the display. Aligning the movement and control directions provides more intuitive control by the user and improves hand-eye coordination, which in turn simplifies the interaction with the system, minimizes the possibility of perforation or damage to tissue during the intervention procedure, shortens the procedure time, and reduces radiation exposure (if x-ray imaging is involved).

[0070] To align the movement and control directions, the current image is aligned along the movement axis x, for example, to provide an optimal viewing orientation of the interventional device movement such that the directional movement of the interventional device on the display coincides with the control movement of the input device on the control console. M , y M The current image may be reoriented relative to the display by rotating the imager (e.g., imager 160) itself, which is used to acquire the current image. If the current image is rotated, a processor (e.g., processor 120) may apply a rotation operation to the raw image data and create a new copy of the current image including the rotated raw image data. The rotated current image is then rendered on the display. Digital images are typically stored as a matrix of elements, e.g., pixels, in which case a rotation matrix is ​​applied to the matrix of elements to result in a rotated current image.

[0071] Instead, the input device is adapted to provide optimal input coordinates to the robot controller, e.g., along a control axis x such that the control direction of the input device coincides with the movement direction of the intervention device on the display. C, y may be reoriented relative to the control device. For example, the functionality of the control console (e.g., input device API) may be dynamically reprogrammed so that the user inputs an input in the left control direction to match the displayed left movement direction of the interventional device, as shown in FIG. 1C. In this case, the processor changes the assignment of the physical control elements of the input device on the control console. A single vector of numerical values ​​may define the behavior of each control element. For example, a thumb stick on the control console that moves the interventional device along the x-axis may be defined as the vector [1,0,0], and a thumb stick that moves along the y-axis may be defined as the vector [0,1,0]. To reorient the control elements, each of the vectors may be rotated via a rotation matrix, resulting in a new vector corresponding to the different orientation. The rotated vectors are then transferred from the processor to the robot controller.

[0072] In embodiments, the user may be alerted to a reorientation of the current image relative to the display or the input device relative to the control console, for example, by a tactile or auditory indicator. Similarly, the user may be alerted to specific interim steps of a reorientation of the current image relative to the display or the input device relative to the control console, such as, for example, rotations of the image or input device in 30 or 45 degree increments.

[0073] In one embodiment, the interventional device in its original orientation may be simultaneously displayed at a reduced size, e.g., as a picture-in-picture, while the current image is reoriented to compensate for the mismatch. Also, in one embodiment, the current image may be continuously or incrementally reoriented (e.g., rotated) to align the movement direction of the interventional device with the control direction of the input device. For example, the current image may be reoriented in discrete angular steps, such as 45 degrees or 90 degrees.

[0074] The steps shown in Fig. 3 are then repeated during the intervention procedure, thereby providing the user with real-time feedback when he uses the control console to control the movement of the interventional device. That is, subsequent images of the interventional device (which may or may not be the next consecutive image) are sequentially displayed as the current image on the display. The movement direction of the interventional device on the display and the control direction of the input device on the control console are determined for each subsequent image. The movement direction and the control direction are aligned, if necessary, by respectively compensating for the estimated discrepancy between them. Thus, the user is provided with real-time feedback that allows intuitive operation of the interventional device when viewing the display throughout the intervention procedure.

[0075] In one embodiment, the alignment of the interventional device movement direction in the image with the control direction of the input device on the control console and compensating for the discrepancy between the two may be based on vascular roadmapping with contrast injection in the anatomical structure, where digital subtraction angiography (DSA) images and fluoroscopy images are stacked along the channel dimension and simultaneously feed into a first neural network model during training to estimate the interventional device movement direction and a second neural network model during training to estimate the discrepancy between the interventional device movement direction and the control direction of the input device.

[0076] In one embodiment, a third neural network model can be trained to learn upcoming workflow and interventional device changes. The current image can then be automatically rotated relative to the display to show the next appropriate orientation of the interventional device to be achieved in the upcoming step so that the movement direction matches the control direction. Transitions between interventional device orientations can be triggered by the location and / or shape of the interventional device relative to the anatomy for a given task. The third neural network model can similarly learn future optimal rotations using pairs of previous image data and corresponding control inputs. The robot controller automatically applies the optimal rotation to the current image in the display.

[0077] In one embodiment, a fourth neural network model can be trained to learn the relationship between the articulation of the robotic interventional device and the articulation of the input device. The fourth neural network can then estimate the future shape of the interventional device in the image by applying data indicative of the movement of the input device.

[0078] As described above with reference to block S314 of Fig. 3, the movement direction of the interventional device in the current image on the display may be estimated according to various embodiments. The estimated movement direction may then be used in estimating a mismatch between the movement direction of the interventional device and a control direction applied to an input device of the control console (block S315) and in aligning the movement direction of the interventional device on the display with the control direction of the input device to compensate for the estimated mismatch (block S316).

[0079] In this context, Fig. 4 shows exemplary display images in which the movement direction of the interventional device in each image is estimated by inferring the surrounding tissue of the anatomical structure and the shape and position of the interventional device from the current image and recent past images of the subject, and Fig. 5 shows exemplary display images in which the movement direction of the interventional device in the current image is estimated by applying a first neural network model to obtain future motion vectors for predicting the corresponding next image showing the future direction of the movement of the interventional device, according to a representative embodiment. In particular, Figs. 4 and 5 show mock images of the interventional device only for clarity. However, it is understood that in reality, the images also include the surrounding tissue of the anatomical structure, which is also used to infer the shape and position of the interventional device and provide sufficient context to train the first neural network, as described above.

[0080] With reference to FIG. 4, the overall shape or distal segment (or alternatively the proximal segment) of the interventional device may be used to set the optimal orientation. The top row shows four successively acquired current images of the interventional device with an arrow indicating the movement direction of the distal segment of the interventional device at the time the corresponding current image was acquired. The bottom row shows four successively acquired current images of the interventional device with an adjustment (e.g., rotation) of the current image to align the movement direction of the interventional device with the corresponding control direction of the input device. As shown, the arrows indicating the movement direction of the distal segment of the interventional device in the bottom row of images all point in the same direction (upward), indicating that the control direction of the input device is also upward, e.g., relative to the control console as shown in FIGS. 1A and 1B.

[0081] Referring to FIG. 5, the top row shows a current image of the interventional device. The middle row shows an estimated future image of the interventional device determined using the first neural network model, where the estimated future image shows a predicted progression of the interventional device from right to left. The bottom row shows a predicted future motion vector for the corresponding estimated future image, which indicates the future movement direction of the interventional device, output by the first neural network model. Specifically, the bottom row presents an overlay of the current image over the estimated future image, showing the respective further motion vector. As shown, the future motion vectors get longer into the future to predict. The future motion vectors can be used to determine an appropriate adjustment (e.g., rotation angle) for the displayed image so that the movement direction in the image aligns with the control direction of the input device.

[0082] More specifically, in the illustrated example, the direction of each future motion vector in the bottom row of FIG. 5 determines the angle by which the current image in the top row is rotated. The length of each future motion vector indicates the number of future steps that need to be completed to fully achieve the desired rotation. The future steps may be determined, for example, based on the time and frame rate of image acquisition. For example, an X-ray imaging device such as a mobile C-arm scanner may acquire images at a frame rate of 5 to 15 images per second. By predicting multiple future movement directions, the system may filter noisy data in time and use the mean or median of the future motion vectors to select optimal parameters for the displayed image.

[0083] 6A illustrates an exemplary control console for controlling directional movement of an interventional device and an exemplary display image of the interventional device as viewed by a user straight ahead, and FIG. 6B illustrates an exemplary control console for controlling directional movement of an interventional device and an exemplary display image of the interventional device as viewed by a user at an offset angle, with matching orientations, according to a representative embodiment. In particular, FIGS. 6A and 6B illustrate a situation in which the physical location and orientation of the display 124 in the examination room changes relative to the user 170. As the location and orientation of the display 124 changes, the rendering of the image 125 on the display 124 also changes, and the control direction D of the input device 145 also changes. C and the direction of movement D of the intervention device 146 M Discontinues the previous alignment of.

[0084] Referring to FIG. 6A, the display 124 is displayed in a direction of gaze D G 1. In this configuration, the control direction D of the input device 145 on the control console 143 is shown directly in front of the user 170, so that the user views the image 125 by looking straight ahead, as indicated by C is the direction of movement D of the interventional device 146 in the image 125 M Assume that the control direction D of the input device 145 is reoriented to match the control direction D of the input device 145. C and the direction of movement D of the intervention device 146 Mmay correspond to the one originally implemented, or may be one or more control directions D according to the above-mentioned embodiments. C Or movement direction D M may be adjusted to match the results of reorienting

[0085] 6B, the display 124 is shown in a different position and orientation offset to the right of the user 170, prompting the user 170 to move in a new gaze direction D G In order to see the image 125, the user is forced to move his / her head to the right, as shown by the image 125 in FIG. M appears to be tilted downwards, and thus the control direction D of the input device 145 C is no longer in the direction of movement D M 3, the orientation of the input device 145 is adjusted relative to the control console 143 to align the direction of movement D of the interventional device 146 in the image 125 as seen on the display 124 at various positions and orientations. M and the control direction D of the input device 145 on the control console 143. C This compensates for discrepancies between

[0086] According to various embodiments of the present disclosure, the methods described herein may be implemented using a hardware computer system executing a software program stored on a non-transitory storage medium. Furthermore, in exemplary non-limiting embodiments, implementations may include distributed processing, element / object distributed processing, and parallel processing. A virtual computer system process may implement one or more of the methods or functions described herein, and a processor described herein may be used to support a virtual processing environment.

[0087] Although estimating and visualizing a trajectory of a robotically controlled interventional device on a display has been described with reference to exemplary embodiments, it being understood that the words used are words of description and illustration, rather than words of limitation. Changes may be made within the scope of the appended claims as presently described and as amended, without departing from the scope and spirit of the embodiments. Although estimating and visualizing a trajectory of a robotically controlled interventional device on a display has been described with reference to particular means, materials, and embodiments, it is not intended to be limited to the details disclosed, but rather, estimating and visualizing a trajectory of a robotically controlled interventional device on a display extends to all functionally equivalent structures, methods, and uses as are within the scope of the appended claims.

[0088] The description of the embodiments described herein is intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the present disclosure described herein. Many other embodiments may be apparent to those skilled in the art upon review of the present disclosure. Other embodiments may be utilized and derived from the present disclosure, such that structural and logical substitutions and modifications may be made without departing from the scope of the present disclosure. In addition, the illustrations are merely representational and may not be drawn to scale. Certain proportions in the figures may be exaggerated and other proportions may be minimized. Thus, the present disclosure and drawings should be considered illustrative rather than limiting.

[0089] One or more embodiments of the present disclosure may be referred to herein, individually and / or collectively, by the term "invention" merely for convenience and without any intention to spontaneously limit the scope of the present application to any particular invention or inventive concept. Furthermore, although specific embodiments have been illustrated and described herein, it should be understood that any subsequent configurations designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. The present disclosure is intended to cover any and all subsequent adaptations or modifications of the various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of the description.

[0090] The Abstract of the Disclosure is provided for purposes of compliance with 37 CFR §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0091] The foregoing description of the disclosed embodiments is provided to enable those skilled in the art to practice the concepts described in this disclosure. Accordingly, the subject matter disclosed above should be considered as illustrative and not limiting, and the appended claims are intended to encompass all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent permitted by law, the scope of the present disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the foregoing detailed description.

Claims

1. 1. A system for displaying and controlling the advancement of an interventional device configured for insertion into a target anatomy, comprising: (i) at least one processor coupled to a display; and (ii) a user interface for providing control inputs for controlling movement of the interventional device; reading a default coordinate system associated with the user interface; receiving current image data of a current image of the interventional device in the anatomical structure that is or is to be displayed on a display, the current image indicating a current position of the interventional device; receiving a control input from the user interface for controlling movement of the interventional device from the current position, the control input representing a control direction in the predetermined coordinate system of the user interface; estimating, from at least the current image data, a direction of movement of the interventional device based on the control input; estimating a discrepancy between the movement direction and the control direction of the interventional device; determining a change in orientation of the current image displayed or to be displayed or a change in orientation of a coordinate system of the user interface so as to align the direction of movement of the interventional device in the current image with the control direction of the user interface; performing said change in orientation; the at least one processor configured to A system having:

2. 2. The system of claim 1, wherein determining a change in orientation of the current image to be displayed or to be displayed comprises rotating the current image to be displayed or to be displayed until the movement direction of the interventional device to be displayed or to be displayed aligns with the control direction.

3. 2. The system of claim 1, wherein determining a change in orientation of the coordinate system of the user interface comprises controlling a change in functionality of the user interface to match the direction of movement of the interventional device in which a control input corresponding to the control direction is displayed or to be displayed.

4. estimating the direction of movement of the interventional device comprises: estimating a shape of the interventional device and surrounding tissue of the anatomical structure from the current image and a plurality of recent past images; estimating the direction of movement of the interventional device based on the shape of the interventional device and the surrounding tissue of the anatomical structure; The system of claim 1 , comprising:

5. estimating the direction of movement of the interventional device comprises: establishing a motion vector for the interventional device using a plurality of recent past images and corresponding control inputs, the motion vector representing a direction and magnitude of displacement of the interventional device as it moves through the anatomical structure shown in the plurality of recent past images; predicting a future motion vector in a corresponding next image that indicates a future direction of movement of the interventional device using a first neural network model, the length of the future motion vector indicating the number of future frames of the next image that are required for the movement of the interventional device to be fully realized; estimating the direction of movement of the interventional device based on the predicted future motion vector; The system of claim 1 , comprising:

6. The at least one processor First, train the first neural network model using motion vectors and corresponding control inputs associated with a plurality of training images, wherein the first neural network model includes a recursive convolutional layer or a transformer architecture; The system of claim 5 further configured to:

7. The at least one processor first, using the current image, the plurality of recent past images and corresponding control inputs, and the estimated movement direction of the interventional device, training a second neural network model to estimate a discrepancy between the movement direction of the interventional device and the control direction of the input device of the control console; The system of claim 5 further configured to:

8. The system of claim 7 , wherein each of the first neural network model and the second neural network model is supervised or self-supervised.

9. The system of claim 7 , wherein each of the first neural network model and the second neural network model is unsupervised.

10. 2. The system of claim 1, further comprising: (i) a user interface coupled to the at least one processor for controlling movement of the interventional device based on control inputs associated with the stored predefined coordinate system; and / or (ii) a display coupled to the at least one processor and configured to display an image of the interventional device within the anatomical structure of the subject.

11. 11. The system of claim 10, wherein the user interface comprises a control console having an input device operable by a user to control movement of the interventional device, the input device optionally being a joystick or thumbstick.

12. The at least one processor determining the control direction of the input device relative to the control console based on the control input; The system of claim 1 further configured to:

13. an imaging system configured to acquire the current image of the anatomical structure; a robot controller configured to enable control of the robot according to the control inputs provided via the user interface; The system of claim 1 further comprising:

14. The system of claim 1 , wherein estimating the direction of movement of the interventional device comprises predicting future movement of the interventional device.

15. The system of claim 1 , wherein the direction of movement of the interventional device is estimated based on movement of the interventional device in a plurality of recent past images.

16. 1. A method for displaying and controlling the progression of an interventional device configured for insertion into a target anatomy, comprising: receiving current image data of the interventional device in the anatomy of the subject, the current image being or to be displayed on a display and indicating a current position of the interventional device; receiving a control input from a user interface for controlling movement of the interventional device from the current position, the control input representing a control direction in an aligned predefined coordinate system of the user interface; estimating, from at least the current image data, a direction of movement of the interventional device based on the control input; estimating a discrepancy between the movement direction of the interventional device and the control direction; determining a change in orientation of the current image or a change in orientation of a coordinate system of the user interface to align the direction of movement of the interventional device in the current image and the control direction of the user interface; performing said change in orientation; A method having the following.

17. 1. A non-transitory computer-readable medium storing instructions for displaying and controlling advancement of an interventional device configured for insertion into a target anatomical structure, the instructions, when executed by at least one processor configured to be coupled to (i) a display and (ii) a user interface for providing control inputs for controlling movement of the interventional device, causing the at least one processor to: reading a predefined coordinate system associated with the user interface; receiving current image data of the interventional device within the anatomical structure to be displayed or to be displayed on a display, the current image indicating a current position of the interventional device; receiving a control input from the user interface for controlling movement of the interventional device from the current position, the control input representing a control direction in the predetermined coordinate system of the user interface; estimating a direction of movement of the interventional device based on the control input from at least the current image data; estimating a discrepancy between the movement direction and the control direction of the interventional device; determining a change in orientation of the current image or a change in orientation of a coordinate system of the user interface to align the direction of movement of the interventional device in the current image and the control direction of the user interface; effecting said change in orientation; Non-transitory computer-readable medium.

Images