Systems and methods for estimating and visualizing the trajectory of a robotically controlled interventional device - Patents.com

Abstract

A method and system that allows for estimating and visualizing a trajectory of an interventional device configured to be robotically guided and inserted into an anatomical structure of a subject, the method includes training a model for predicting a trajectory of the interventional device based on training data from a previous image and corresponding control inputs, receiving image data from an image showing a current position of the interventional device, receiving untriggered control inputs for controlling the robot to guide future movement of the interventional device, predicting a trajectory of the interventional device by applying the image data and the untriggered control inputs to the training model, displaying the predicted trajectory of the interventional device overlaid on an image of the anatomical structure, and if the predicted trajectory is determined to be acceptable, triggering the untriggered control inputs to control the robot to guide movement of the interventional device according to the triggered control inputs.

Description

[Technical field]

[0001] Robotic systems used in interventional procedures (such as endovascular procedures) are operated by a user manipulating a robot controller that includes an interface device such as a joystick controller. The robotic system controls the movement of various interventional devices, such as catheters and guidewires. The motion and behavior of the interventional devices in a subject (patient) depends on the inputs the user triggers from the robot controller, as well as the shape, properties, and location of the interventional devices relative to the subject's anatomy. [Background technology]

[0002] In robotic-assisted interventional procedures, steering a bendable interventional device through highly curvatured, highly branched, and tortuous anatomical structures (e.g., blood vessels) can be difficult and complex, potentially resulting in multiple failed attempts leading to damage to the anatomical structures (e.g., bleeding), increased procedure time, increased imaging radiation exposure to the subject, and / or the need for replacement of the interventional device. Conventional solutions are unable to visualize or otherwise predict the potential trajectory of the interventional device given various inputs to the robotic system. As a result, users may apply suboptimal inputs, which can result in multiple failed attempts. Summary of the Invention [Problem to be solved by the invention]

[0003] Therefore, there is a need for a robotic system that predicts the trajectory of an interventional device during an interventional procedure based on control inputs by a user, and provides a display showing a dynamic line indicating where the control inputs will position the interventional device before actually moving the interventional device. [Means for solving the problem]

[0004] In an exemplary embodiment, a system for estimating and visualizing a trajectory of an interventional device configured for robotic guided insertion into an anatomy of a subject is provided, the system including at least one processor and a non-transitory memory for storing machine-executable instructions. The machine-executable instructions, when executed by at least one processor, cause the at least one processor to: receive image data from a current image of the anatomical structure, the current image indicating a current position of an interventional device relative to the anatomical structure; receive at least one untriggered control input for controlling a robot to guide future movement of the interventional device from the current position; predict at least one trajectory of the interventional device in the current image by providing the image data and the at least one untriggered control input to a model, the model being configured to predict the trajectory of the interventional device based on a first input of image data of the interventional device in the anatomical structure and a second input of a corresponding control input to the robot to guide movement of the interventional device as indicated in the image data of the first input; and provide a trigger command to trigger the at least one untriggered control input to control the robot to guide movement of the interventional device in accordance with the at least one triggered control input.

[0005] In an exemplary embodiment, a system for estimating and visualizing a trajectory of an interventional device configured to be guided by a robot and inserted into an anatomical structure of a subject is provided. The system includes at least one processor and a non-transitory memory for storing machine-executable instructions. The machine-executable instructions, when executed by the at least one processor, cause the at least one processor to: receive a model configured to predict a trajectory of an interventional device from an image of the interventional device within the anatomical structure and corresponding control inputs to the robot for guiding movement of the interventional device as shown in a previous image; receive image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure; receive untriggered control inputs for controlling the robot to guide future movement of the interventional device from the current position; and receive the image data and The trajectory of the interventional device in the current image is predicted by applying the untriggered control inputs to the model, the predicted trajectory of the interventional device is overlaid on the current image of the anatomical structure for a user to determine whether the predicted trajectory is acceptable, if the predicted trajectory is determined to be acceptable, a trigger command is received to trigger the untriggered control inputs to control the robot to guide movement of the interventional device in accordance with the triggered control inputs, and if the predicted trajectory is determined to be unacceptable, a new untriggered control input is received to control the robot to alternatively guide future movement of the interventional device from the current position.

[0006] In another representative embodiment, a method is provided for estimating and visualizing a trajectory of an interventional device configured for robotic guided insertion into a subject's anatomy. The method includes the steps of training a neural network model in predicting a trajectory of an interventional device based on training data from a previous image of the interventional device and corresponding control inputs to the robot for guiding movement of the interventional device as shown in the previous image; receiving image data from at least one image of an anatomical structure, the at least one image showing a current position of the interventional device relative to the anatomical structure; receiving an untriggered control input for controlling the robot to guide future movement of the interventional device from the current position; predicting a trajectory of the interventional device in the at least one image by applying the image data and the untriggered control input to the trained neural network model; overlaying the predicted trajectory of the interventional device on the at least one image of the anatomical structure; if the predicted trajectory is determined to be acceptable, triggering the untriggered control input to control the robot to guide movement of the interventional device in accordance with the triggered control input; and if the predicted trajectory is determined to be unacceptable, receiving a new untriggered control input for controlling the robot to alternatively guide future movement of the interventional device from the current position.

[0007] In another representative embodiment, a system for determining and visualizing a trajectory of an interventional device configured to be guided by a robot and inserted into an anatomical structure of a subject is provided. The system includes at least one processor and a non-transitory memory for storing machine-executable instructions. The machine-executable instructions, when executed by the at least one processor, cause the at least one processor to: receive previous image data from a previous image of the anatomical structure showing the interventional device and the anatomical structure; determine previous position data of the interventional device from the previous image data to provide a position and trajectory of the interventional device in the previous image; receive control inputs for controlling the robot to guide movement of the interventional device from the previous position; calculate a displacement of the interventional device from the previous position based on the received control inputs applied to the previous position data; transform the calculated displacement of the interventional device into image coordinates using a calibration of the previous image; and overlay a predicted trajectory of the interventional device on the previous image using the image coordinates.

[0008] In another representative embodiment, a system for determining and visualizing a trajectory of an interventional device configured to be guided by a robot and inserted into an anatomical structure of a subject is provided. The system includes at least one processor and a non-transitory memory for storing machine-executable instructions. The machine-executable instructions, when executed by the at least one processor, cause the at least one processor to: train a neural network model with respect to determining control inputs for controlling the robot to guide the interventional device along a given predicted trajectory; receive current image data from a current image of the anatomical structure showing the interventional device and the anatomical structure; display a predicted trajectory of the interventional device overlaid on the current image, the predicted trajectory of the interventional device being predicted by applying the current image data and the control inputs to the trained neural network model; receive instructions to adjust the displayed predicted trajectory to a desired position; determine a control input corresponding to a desired position of the predicted trajectory by applying image coordinates of the predicted trajectory at the desired position to the neural network model; and provide the determined control input to a robot controller for controlling the robot to guide the interventional device along the desired predicted trajectory.

[0009] In another representative embodiment, a system for estimating and visualizing a trajectory of an interventional device configured for robotic guided insertion into an anatomy of a subject is provided, the system including at least one processor and a non-transitory memory for storing machine-executable instructions.The machine executable instructions, when executed by the at least one processor, cause the at least one processor to: receive a model configured to predict a trajectory of an interventional device based on data from an image of the interventional device within the anatomical structure and a corresponding control input to the robot for guiding movement of the interventional device as shown in the image; receive image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure, the image data being collected on a discontinuous basis; determine current position data of the interventional device from the image data; determine current control and state data of the robot using the robotics data from the robotic system for controlling the robot; receive a triggered control input to control the robot to guide movement of the interventional device from the current position; and receive image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure, the image data being collected on a discontinuous basis. and predicting a trajectory of the interventional device in the current image by applying the new image data and the triggered control inputs to the model; overlaying the predicted trajectory of the interventional device on the current image of the anatomical structure to show an actual trajectory of the interventional device without receiving additional image data; determining whether new image data for a new current image has been received; and if new image data has been received, determining new current position data of the interventional device from the new image data, determining new current control and state data of the robot using robotics data from a robotic system for controlling the robot, and receiving triggered control inputs for controlling the robot to guide a current movement of the interventional device from the new current position, iteratively; and predicting a trajectory of the interventional device in the new current image by applying the new image data and the triggered control inputs to the model.

[0010] The example embodiments are best understood when the following detailed description is read in conjunction with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions have been arbitrarily increased or decreased for clarity of illustration. Where applicable and practical, like reference numerals refer to like elements. [Brief description of the drawings]

[0011] [Figure 1] 1 illustrates an exemplary interventional device with a predicted trajectory and associated uncertainty margin on the display, according to a representative embodiment. [Diagram 2] FIG. 1 is a simplified block diagram of a system for estimating and visualizing the trajectory of a robotically controlled interventional device on a display including a graphical user interface (GUI), according to a representative embodiment. [Diagram 3] FIG. 1 is a simplified block diagram illustrating an example implementation for training an encoder-decoder neural network architecture according to a representative embodiment. [Figure 4] FIG. 1 is a flow diagram illustrating a method for estimating and visualizing a trajectory of a robotically controlled interventional device on a display including a GUI, according to a representative embodiment. [Diagram 5] FIG. 1 is a flow diagram of a method for determining and visualizing a trajectory of a robotically controlled interventional device on a display including a GUI, according to another representative embodiment. [Figure 6] FIG. 13 is a flow diagram of a method for controlling a robotically controlled interventional device by adjusting a visualized trajectory of the interventional device on a display including a GUI, according to another representative embodiment. [Figure 7] FIG. 1 is a simplified block diagram illustrating an example implementation for training an encoder-decoder neural network architecture according to a representative embodiment. [Figure 8]FIG. 1 is a simplified block diagram illustrating another example implementation for training an encoder-decoder neural network architecture according to a representative embodiment. [Figure 9] FIG. 1 is a simplified block diagram of an encoder-decoder neural network model configured to predict a device trajectory in a dimension-preserving architecture, according to a representative embodiment. [Figure 10] FIG. 1 is a simplified block diagram of an encoder-decoder neural network model configured to predict a device trajectory with a dimensionally variable architecture, according to a representative embodiment. [Figure 11] FIG. 1 is a flow diagram illustrating a method for estimating and visualizing the trajectory of a robotically controlled interventional device using discontinuous imaging, according to a representative embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted so as not to obscure the description of the representative embodiments. However, systems, devices, materials, and methods within the purview of those skilled in the art may be used in accordance with the representative embodiments within the scope of the present teachings. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Defined terms are in addition to the technical and scientific meaning of the defined terms as commonly understood and accepted in the art of the present teachings.

[0013] In this specification, terms such as "first," "second," and "third" are used to describe various elements or components, but it should be understood that these elements or components are not limited by these terms. These terms are used only to distinguish one element or component from another. Thus, a first element or component described below may be referred to as a second element or component without departing from the teachings of the inventive concept.

[0014] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in this specification and the appended claims, singular terms are intended to include both the singular and the plural, unless the context clearly indicates otherwise. Furthermore, the term "comprises" and / or similar terms specify the presence of stated features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0015] Unless otherwise noted, when an element or component is said to be "connected," "coupled," or "adjacent" to another element or component, it is understood that the element or component may be directly connected or coupled to the other element or component, or there may be intervening elements or components present. That is, these and similar terms encompass the cases where one or more intermediate elements or components may be used to connect the two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this only encompasses the cases where the two elements or components are connected to one another with no intermediate or intervening elements or components.

[0016] The present disclosure is intended to provide one or more of the advantages specifically set forth below through one or more of its various aspects, embodiments, and / or specific features, subcomponents. For purposes of explanation and not limitation, example embodiments disclosing specific details are described to provide a thorough understanding of the embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are within the scope of the appended claims. Furthermore, descriptions of well-known devices and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and devices are within the scope of the present disclosure.

[0017] In general, various embodiments described herein provide systems and methods that allow a user (interventional physician) of a robotic system to visualize the future (predicted) trajectory of a robotically controlled interventional device based on preliminary (intended) control inputs to the robotic system before the control inputs are applied or executed. Thus, the user can adjust the preliminary control inputs based on the predicted trajectory before triggering the interventional device movement. Predicting the trajectory based on the preliminary inputs can reduce the number of failed attempts, reduce the risk of tissue puncture or other damage, reduce radiation exposure, and generally improve the usability of the robotic system. The robot controller estimates and visualizes the predicted trajectory of the robotically controlled interventional device before the preliminary inputs are triggered in an uncalibrated robot-patient setup. The estimation and visualization are based on planned inputs by the user (without activating the robot), and the predicted trajectory of the interventional device is overlaid on the most recent fluoroscopic image.

[0018] FIG. 1 illustrates an exemplary interventional device with a predicted trajectory and associated uncertainty margin on the display, according to a representative embodiment.

[0019] 1, the display 124 shows a fluoroscopic image of an interventional device 146 configured for insertion into a subject's anatomy 155, such as a blood vessel or artery. The interventional device 146 is guided by a robotic controller and a robot, such as a robotic controller 142 and a robot 144 described below with reference to FIG. 2. The interventional device 146 is any compatible medical instrument capable of robotic control, such as a catheter, a guidewire, an endoscope, a camera, a surgical tool, etc. The image on the left shows a current position of the interventional device 146 within the anatomy 155. The image on the right shows a projected trajectory of the interventional device 146 within the anatomy 155 based on control inputs for controlling the robot to guide the interventional device 146, if the control inputs are maintained. The control inputs include user inputs provided to the robot via a user interface and robotic controller to steer and otherwise manipulate the robot within the anatomy 155, and status information provided by the robotic controller to indicate the status of the robotic system in response to the user inputs. For example, control inputs are not triggered, where the user inputs a desired user input but does not yet trigger the robot controller to implement the entered control input. The user inputs include directional and velocity control signals for the robot to navigate the interventional device 146, and the state information includes the physical positioning of the robot as well as kinematics for the robot's articulations, rotations, translations, velocities, accelerations, etc.

[0020] In the illustrated embodiment, the predicted trajectory is indicated by a centerline 141, and the estimated uncertainty associated with the predicted trajectory is indicated by an uncertainty margin indicated by outer lines 143 and 145 on either side of the centerline 141. The centerline 141 extends longitudinally from the distal end of the interventional device 146 in its current position along an estimated path that the interventional device 146 would follow if a control input by the user were maintained (i.e., triggered). The outer lines 143 and 145 deviate more and more from the centerline 141 the further the centerline 141 extends from the current position of the interventional device 146. Thus, the outer lines 143 and 145 indicate that the further the predicted trajectory extends from the current position, the more uncertainty there is in the prediction. Alternatively or additionally, the uncertainty margin may be indicated by color-coded pixels that define a margin along the centerline of the predicted trajectory, for example within the boundaries of the outer lines 134 and 145. The outer lines 143 and 145 may or may not be displayed with colored pixels.

[0021] The display 124 is continually updated as additional images are acquired and / or as the user changes or refines the user inputs. The predicted trajectory and associated uncertainty are determined using a neural network module that has been trained using previously acquired images of the interventional device 146 along with corresponding previous control inputs that were used to position the interventional device 146 as shown in the previous images, as described below.

[0022] FIG. 2 is a simplified block diagram of a system for estimating and visualizing the trajectory of a robotically controlled interventional device on a display including a GUI, according to a representative embodiment.

[0023] 2, the system 100 includes a workstation 105 for implementing and / or managing the processes described herein with respect to controlling the movement of an interventional device 146 within an anatomical structure 155 of a subject (patient) 150. The workstation 105 includes one or more processors, represented by processor 120, one or more memories, represented by memory 130, a user interface 122, and a display 124. The processor 120 interfaces with a robotic system 140 via a controller interface (not shown). The robotic system 140 includes a robot controller 142 and a robot 144. The robot controller 142 is configured to control the movement of the robot 144 in response to user input received via the user interface 122 or another robot controlled interface (not shown). The robot 144 is attached to or integrated with the interventional device 146. The robot 144 includes segments, joints, servo motors, and other control features that can move and position the interventional device 146 with multiple degrees of freedom (DOF) in response to control signals received from the robot controller 142. In the illustrated embodiment, the robot controller 142 is shown separate from the processor 120 in the workstation 105 for illustrative purposes. However, it will be understood that all or a portion of the functionality of the robot controller 142 may be incorporated into the processor 120 or vice versa without departing from the scope of the present teachings.

[0024] Processor 120 also interfaces with imaging device 160 via an imaging interface (not shown). Imaging device 160 may be any of a variety of types of medical imaging devices / modalities, such as a fixed or mobile C-arm fluoroscopy system, an X-ray imaging device, a computed tomography (CT) scanning device, a magnetic resonance (MR) imaging device, a positron emission tomography (PET) scanning device, or an ultrasound device. Imaging device 160 may include a single or multiple imaging modalities.

[0025] The memory 130 stores instructions executable by the processor 120. When the instructions are executed, the processor 120 performs one or more processes for estimating and visualizing a trajectory of a robotically controlled interventional device 146 on the display 124. For purposes of illustration, the memory 130 is shown as including software modules, each of which contains instructions corresponding to an associated capability of the system 100.

[0026] Processor 120 represents one or more processing devices, implemented using hardware, software, firmware, hardwired logic circuits, or combinations thereof, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), general purpose computers, central processing units, graphical processing units, computer processors, microprocessors, microcontrollers, state machines, programmable logic devices, or combinations thereof. Any processing unit or processor herein may include multiple processors, parallel processors, or both. Multiple processors may be included or combined in a single device or multiple devices. As used herein, the term "processor" encompasses electronic components capable of executing programs or machine-executable instructions. A processor may also refer to a collection of processors in a single computer system, or a collection of processors distributed across multiple computer systems, such as in a cloud-based or other multi-site application. A program comprises software instructions executed by one or more processors, either in the same computing device or distributed among multiple computing devices.

[0027] Memory 130 includes main memory and / or static memory. Such memories may communicate with each other and with processor 120 via one or more buses. Memory 130 may be implemented with any number, type, and combination of random access memories (RAMs) and read-only memories (ROMs), etc., and may store various types of information, such as software algorithms, artificial intelligence (AI) machine learning models, and computer programs, all executable by processor 120. The various types of ROMs and RAMs may include any number, type, and combination of computer-readable storage media, such as disk drives, flash memory, electrically programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), registers, hard disks, removable disks, tapes, compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), floppy disks, Blu-ray disks, universal serial bus (USB) drives, or any other form of storage media known in the art. Memory 130 is a tangible storage medium for storing data and executable software instructions, and is non-transient while the software instructions are stored therein. As used herein, the term "non-transient" is to be construed as a characteristic of a state that persists for a period of time, rather than an immutable characteristic of a state. The term "non-transient" specifically negates momentary characteristics, such as carrier waves or signals, or other forms of characteristics that exist only temporarily at any time and in any place. Memory 130 stores software instructions and / or computer readable code that enable the performance of various functions. Memory 130 may be secure and / or encrypted, or unsecure and / or unencrypted.

[0028] The system 100 also includes a database 112 for storing information used by the various software modules in the memory 130. For example, the database 112 includes image data from previously acquired images of the subject 150 and / or other similarly situated subjects having the same or similar interventional procedure as the subject 150. The stored image data can be used to train AI machine learning models, such as neural network models, as described below. The database 112 can be implemented with any number and combination of RAMs and ROMs, etc. The various types of ROMs and RAMs include any number, type, and combination of computer readable storage media, such as disk drives, flash memory, EPROMs, EEPROMs, registers, hard disks, removable disks, tapes, CD-ROMs, DVDs, floppy disks, Blu-ray disks, USB drives, or any other form of storage media known in the art. The database 112 includes tangible storage media for storing data and executable software instructions, and is non-transitory while the data and software instructions are stored. Database 112 may be secure and / or encrypted, or non-secure and / or unencrypted. For purposes of illustration, database 112 is shown as a separate storage medium, but it will be understood that it may be combined with or included in memory 130 without departing from the scope of the present teachings.

[0029] Processor 120 includes or has access to an artificial intelligence (AI) engine. The AI ​​engine is implemented as software that provides artificial intelligence (such as neural network models) and applies machine learning as described herein. The AI ​​engine may reside in various components in addition to or other than processor 120, such as memory 130, an external server, and / or the cloud. For example, when the AI ​​engine is implemented in the cloud, such as a data center, the AI ​​engine may be connected to processor 120 over the Internet using one or more wired or wireless connections.

[0030] The user interface 122 is configured to provide information and data output by the processor 120, memory 130, and / or robot controller 142 to a user and / or to receive information and data input by a user. That is, the user interface 122 allows a user to input data and control or manipulate aspects of the processes described herein. It may also allow the processor 120 to indicate the actions of the user's control or manipulation. All or a portion of the user interface 122 may be implemented by a graphical user interface (GUI), such as a GUI 128 viewable on a display 124, described below. The user interface 122 may include one or more interface devices, such as a mouse, keyboard, trackball, joystick, microphone, video camera, touchpad, touch screen, voice captured by a microphone or video camera, or gesture recognition.

[0031] Display 124 may be a monitor, such as a computer monitor, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT) display, or, for example, an electronic whiteboard. Display 124 includes a screen 126 for viewing an internal image of subject 150, along with various features described herein, to assist the user in reading the image accurately and efficiently, and a GUI 128 that allows the user to interact with the displayed images and features. The user may personalize various features of GUI 128, described below, for example, by creating specific alerts and reminders.

[0032] With reference to the memory 130, various modules store sets of data and instructions executable by the processor 120 to estimate and visualize a predicted trajectory of the robotically controlled interventional device 146. The current image module 131 is configured to receive, process, and display on the display 124 a current (first) image of the anatomy 155 of the subject 150. The current image is the image currently viewed by the user during the interventional procedure. The current image is received in real-time from the imaging device 160 during a current contemporaneous imaging session of the subject 150. Alternatively, the current image may be an image of the subject 150 acquired previously in the current imaging session. The imaging has been paused, for example, to reduce radiation exposure of the subject 150. Similarly, the current image may be retrieved from the database 112, which stores images acquired during a previous imaging session or images acquired earlier in the current imaging session (from single or multiple imaging modalities). The current image is displayed on the screen 126 to allow the user to analyze and navigate the robot 144.

[0033] The control input module 132 is configured to receive user inputs from a user via the user interface 122 for controlling the robot 144 to guide the movement of the interventional device 146, and status information as control inputs via the robot controller 142. According to various embodiments, the control inputs are initially untriggered, and the user inputs the user inputs without being executed by the robot controller 142. In this manner, the processor 120 can predict the effect of the control inputs on the trajectory of the interventional device 146 before the robot controller 142 actually controls the robot 144 to move the interventional device 146 in response to the control inputs, as described below. This allows the user to determine whether an untriggered control input is appropriate before action is taken to implement the control input. For example, it can be determined whether the control input, when executed, will move the interventional device 146 to a desired location by a desired route, avoiding damage to the anatomical structure 155. The control inputs include manipulation instructions such as articulation, rotation, translation, distance, velocity, and acceleration of the robot 144 to move the interventional device 146 within the anatomy 155 .

[0034] The previous images module 133 is configured to receive previous images of the subject 150, for example from the imaging device 160 and / or the database 112. The previous images may include images of the subject 150 collected earlier in the current imaging session or during a previous imaging session involving the same type of interventional procedure. The images may be collected from a single or multiple imaging modalities. In an embodiment, the previous images module 133 may also receive previous images of other similarly situated subjects having the same or similar interventional procedures as the subject 150 having the same type of interventional procedure. The previous images are used to train a neural network model, as described below.

[0035] The previous control input module 134 is configured to receive previous control inputs associated with a previous image of the previous image module 133. That is, for each previous image, the previous control input module 134 receives a corresponding control input that is provided to the robot controller 142 that was used to manipulate the interventional device to the position captured by that previous image. The association between the previous control input and the previous image is maintained.

[0036] The predictive model module 135 is configured to implement a predictive model, such as a neural network model, for predicting a trajectory of the interventional device 146 based on control inputs for controlling the movement of the robot 144 via the robot controller 142. The predictive model can also estimate an uncertainty associated with the predicted trajectory. For example, the predictive model can be displayed on the display 124 with a centerline (e.g., centerline 141) as described above, and an uncertainty margin (e.g., outer lines 143 and 145) indicating the uncertainty as a function of distance from a current known position of the interventional device 146 can be determined. In an embodiment, the uncertainty margin can be extended to a volumetric model, such as a 3-dimensional convolutional neural network (CNN). In various embodiments, all or part of the processes provided by the predictive model module 135 may be implemented by an AI engine, for example, as described above.

[0037] The predictive model module 135 provides training of the predictive model with respect to predicting the trajectory and associated uncertainty of the interventional device 146 based on training data. The training data includes a previous image of the interventional device and corresponding control inputs for guiding the interventional device as shown in the previous image. For example, the training data includes a previous image of the interventional device 146 within the anatomy 155 of the subject 150 during a current imaging session and corresponding control inputs to the robot 144 used to guide the interventional device 146 to the position shown in the previous image. The previous image is obtained from the previous image module 133 and the corresponding control inputs are obtained from the previous control input module 134. Alternatively, the training data may include a previous image and corresponding control inputs from a previous imaging session involving the subject 150 or other similarly situated subjects, as described above. In an embodiment, the training data may also include a current image obtained from the current image module 131 and corresponding control inputs obtained from the control input module 132, allowing for on-the-fly training of the predictive model. It will therefore be understood that the descriptions herein relating to training using a previous image and corresponding control inputs equally apply to training using a current image and corresponding control inputs without departing from the scope of the present teachings.

[0038] Training a predictive model involves capturing the spatial and temporal context of previous images. Spatial context can be captured using convolutional layers. Convolutional layers consist of a sliding window, or kernel, that slides over the input images, represents a matrix of weights that perform element-wise multiplication with the overlapping parts of the input images, and consolidates the results into an output feature map. Temporal context can be captured using temporal connections across layers, such as by using recurrent neural networks (RNNs), long short-term memories (LSTMs), transformers, etc. Training a predictive model can be unsupervised (unsupervised training) or supervised (supervised training), where a collection of images and control inputs serves as the output (labels) of the collection of previous images and control inputs. Thus, during the training phase, the predictive model learns an appropriate representation from retrospective data that uses previous image frames 1, …, n-1 to predict the trajectory of the interventional device at the next m time points (image frames n, …, n+m).

[0039] The previous image and previous control input data may be fed together at the earliest layer of the neural network model. An example of this is described below with reference to FIG. 9. The previous (and current) image data may include, for example, fluoroscopic images or segmentation maps of various devices or anatomical structures in the images. The previous (and current) control input data may include measurements from the user interface 122 and / or the controller console of the robot controller 142, the kinematics of the system in terms of articulation, rotation, and translation, as well as velocity and acceleration, device types and device stacks used by the robot, the number of actuated devices, and robot calibration parameters. Alternatively, the previous image data may be fed at the earliest layer of the neural network model, and the previous control input data may be used at an intermediate or latent layer of the neural network model, which acts as a transformation applied to the representation learned from the previous image frames 1, ..., n-1. An example of this is described below with reference to FIG. 10. In this case, the control input data may be passed through a series of fully connected layers before being merged with the convolutional network. The predicted outputs at future image frames n,...,n+m will generate different trajectories for different robot transformation or control input data. In another implementation, two separate neural networks can be used, one for imaging data and one for robot control data. In this case, the two neural networks share weights or feature maps at some intermediate layers. Training of the neural network model is done iteratively. At each iteration, a batch of corresponding previous images and corresponding previous control input data is fed to the neural network model. Before training, a similarity loss is minimized, such as binary cross entropy or intensity loss, which will be clear to those skilled in the art.

[0040] In an embodiment, the predictive model is trained for predicting the trajectory of the interventional device based on training data from an image of the interventional device and a control input to the robot for guiding the movement of the interventional device corresponding to the image. Training the predictive model includes receiving the training data, inputting at least one image of the interventional device in an anatomical structure to the predictive model, inputting at least one control input to the robot for guiding a subsequent movement of the interventional device, generating a predicted trajectory of the interventional device using the predictive model, and adjusting parameters of the predictive model based on a difference between the predicted trajectory and a corresponding ground truth trajectory from at least one image of the interventional device corresponding to the input robot control input. These steps are repeated until a stopping criterion is met. For example, the stopping criterion is when the difference between the estimated trajectory of the interventional device and the ground truth trajectory of the interventional device is less than a predetermined threshold. Another stopping criterion may be the distance between the tip (distal portion) of the trajectory and the target anatomical landmark (such as the tip of the trajectory near an ostium or aneurysm).

[0041] The predictive model module 135 predicts the trajectory of the interventional device 146 by training the predictive model as well as applying the current images from the current image module 131 and the control inputs from the control input module 132 to the trained predictive model. The output of the predictive model provides the predicted trajectory and the uncertainty of the predicted trajectory as a function of the distance the predicted trajectory extends from the current position of the interventional device 146. The distance or time into the future at which the predicted trajectory is generated is a function of the velocity of the interventional device 146 (per degree of freedom or linked to the current projection). The gain may be adjusted by the user.

[0042] That is, the predictive model module 135 receives image data from a current image of the anatomical structure 155, indicating the current position of the interventional device 146 relative to the anatomical structure 155, and also receives control inputs for controlling the robot 144 to guide future movement of the interventional device 146 from the current position. Because the control inputs are untriggered, the predicted trajectory can be determined before the robot controller 142 actually performs the control inputs. The control inputs include current robot parameters and configuration space, such as joystick inputs, forces, torques, and forward and inverse kinematics. The predictive model module 135 can also input a segmentation map associated with the current image. The predictive model provided by the predictive model module 135 predicts the trajectory of the interventional device 146 in the current image, as well as the associated uncertainty, by applying the image data and the untriggered control inputs to the trained predictive model described above.

[0043] 3 is a simplified block diagram illustrating an example implementation for training an encoder-decoder neural network architecture, according to a representative embodiment. An encoder-decoder neural network architecture is used, where a current image is input to an encoder portion of the neural network, which generates a latent representation that is a reduced dimension of the input image. The encoder and decoder portions may include multiple blocks to capture temporal dependencies, such as RNNs, LSTMs, transformers, etc. Untriggered control inputs can be applied to the latent representation to generate a transformed latent representation, which can be decoded using the decoder portion of the neural network to generate a predicted image that includes a predicted trajectory of the interventional device 146.

[0044] Referring to FIG. 3, a training image 311 is used as a ground truth image to which a known control input C has been applied. During training, for example, a predicted image 312 is compared to the ground truth image 311. The predicted image 312 results from applying the control input C to the current position of the interventional device 146. As described above, the comparison is performed using a loss function 320, and the loss can be used to perform gradient descent to update the neural network parameters over multiple iterations to minimize the loss function calculated over multiple iterations. As an example, a similar concept, the transformation of an auto-encoder, is described by GE Hinton et al. in "Transforming auto-encoders" (ICANN 2011), which is incorporated herein by reference. The predicted trajectory and uncertainty can also be fed back as training data as input to the predictive model module 135 to further train the neural network and aid in predicting subsequent untriggered control input data. Once the predicted trajectory is determined to be acceptable and the untriggered control input is triggered, the resulting trajectory is also fed back to the neural network as the new current image. Additionally, as described above, the current image data and the corresponding current untriggered control inputs are fed back as inputs to the neural network module 135 as training data for on-the-fly training.

[0045] The uncertainty can be estimated, for example, using a Monte Carlo dropout algorithm that approximates Bayesian estimation of a deep Gaussian process. To trigger the dropout, a subset of neurons is switched off during the forward pass. To calculate the uncertainty from the dropout, every received batch of data is passed through the neural network model multiple times (e.g., 10 times). Each time, the dropout algorithm provides a slightly different form of the neural network model that can then result in another trajectory prediction by the neural network model. The results are aggregated to calculate upper and lower bounds on the predicted trajectory. These upper and lower bounds can be visualized on the current image, for example, as outer lines 143 and 145, as described below, to indicate the uncertainty of the predicted trajectory.

[0046] In an embodiment, a roadmap image is incorporated into the predictive model. The roadmap image contains registered anatomical structure (such as blood vessels) information from contrast angiograms and / or pre-treatment imaging. The boundaries of the anatomical structure 155 indicated by the roadmap image affect the expected shape of the interventional device 146. Thus, by incorporating the roadmap image, the predictive model returns a different predicted trajectory and uncertainty output, resulting in a different graphical representation of the predicted trajectory.

[0047] The display processing module 136 is configured to receive the predicted trajectory and associated uncertainty and display the predicted trajectory and associated uncertainty on the display 124, overlaid on a current image of the anatomical structure 155. For example, the display processing module 136 is configured to display the predicted trajectory as a centerline (such as centerline 141) and the associated uncertainty as uncertainty margins (such as outer lines 143 and 145), as described above with reference to FIG. 1. When multiple coaxial interventional devices 146 are present, each interventional device 146 has its own predicted trajectory representation of the corresponding predicted trajectory, and these predicted trajectory representations are essentially coupled with the estimated behavior of both interventional devices 146.

[0048] In the case of in-plane and out-of-plane motion of the interventional device 146, the color scheme of the predicted trajectory can be correlated with the change in magnitude and direction of the shape of the interventional device 146 along the predicted trajectory. For example, if the interventional device 146 is predicted to move towards a detector of the imaging device 160, the shape of the interventional device 146 changes very little in the current image while the robot 144 translates the interventional device 146. In this case, the color of the predicted trajectory changes between a first color (e.g., more blue) where the interventional device 146 is predicted to shape towards the detector, and a second color (e.g., more green) where the interventional device 146 is predicted to shape away from the detector.

[0049] This display allows the user to determine, given the uncertainty margin, whether the predicted trajectory is acceptable for navigation of the interventional device 146. For example, the predicted trajectory is acceptable if it advances the interventional device 146 towards the target while avoiding contact with the inner wall of the anatomical structure 155 and / or if the predicted trajectory substantially matches the desired trajectory of the interventional device 146. If a volumetric model is used, the display processing module 136 displays a projection of the uncertainty margin or a three-dimensional predicted trajectory superimposed on the current image.

[0050] If the predicted trajectory is acceptable, the user triggers the robotic system 140 to move the interventional device 146 according to the control input. That is, the robot controller 142 receives a trigger command, for example from the user interface 122, to activate an untriggered control input, and in response, controls the robot 144 according to the triggered control input to guide the movement of the interventional device 146. Ideally, the robot 144 moves the interventional device 146 along a path that is consistent with the predicted trajectory, but the path may deviate from the predicted trajectory, especially if the uncertainty is relatively high. In an embodiment, image data from an additional image of the anatomical structure 155 can be received from the imaging device 160 after triggering the untriggered input. In this case, after triggering the untriggered input, the actual trajectory of the interventional device 146 is displayed with the predicted trajectory superimposed on the current image on the display 124. Also, the interventional device 146 may be displayed superimposed on the current image while moving the interventional device 146 according to the triggered input. Alternatively, image data from an additional (second) image of the anatomical structure 155 can be received from the imaging device 160 while controlling the current movement of the interventional device 146 according to the triggered control inputs. In this case, the actual trajectory of the interventional device 146 is superimposed on the additional image while controlling the current movement of the interventional device 146.

[0051] Alternatively or additionally, the acceptability of the predicted trajectory is determined automatically, for example, by the processor 120. The shape of the predicted trajectory can be captured, for example, using known image analysis techniques that provide a segmented predicted trajectory, and comparing the segmented predicted trajectory to a desired trajectory for navigating the interventional device 146. For example, the predicted trajectory can be compared to a desired centerline or other datum in the anatomy 155. If the comparison is within a predetermined error range, the processor 120 validates the predicted trajectory. In another example, a user can draw a desired trajectory on the GUI 128 and compare the segmented predicted trajectory to the desired trajectory to determine the acceptability of the predicted trajectory. A control input can then be triggered automatically or by the user upon notification that the trajectory has been validated. In another example, the segmented predicted trajectory can be compared to a predefined set of rules for navigating the interventional device 146. For example, the predicted trajectory is acceptable if it advances the interventional device toward the target while avoiding contact with the inner wall of the anatomy. In another example, the estimated trajectory can be compared to the desired trajectory by comparing curves (e.g., based on thresholds or geodesic distances), parameterizing the curves using polynomials and splines and comparing distances in parameter space, or comparing the distance of the apex of the estimated trajectory to desired anatomical landmarks.

[0052] If the predicted trajectory is not acceptable, the user can adjust the user input of the untriggered control input to the robot controller 142 for controlling the robot 144. This will guide future movement of the interventional device 146 from its current position along a different predicted trajectory. In this case, the control input module 132 receives new user input from the user via the user interface 122 and state information from the robotic system 140 as new control input. Again, this new control input is initially untriggered. The user inputting the user input without it being executed by the robot controller 142 allows the processor 120 to predict the effect of the new control input on the trajectory of the interventional device 146 without the robotic system 140 actually moving the interventional device 146.

[0053] The predictive model module 135 receives the new control inputs along with a current image of the interventional device 146 within the anatomy 155 provided by the current image module 131. The current image may be the same image used to predict the trajectory based on the previous control inputs or a more recently acquired image. The predictive model module 135 applies the new control inputs and the current image to a predictive model. The predictive model outputs a new (adjusted) predicted trajectory of the interventional device 146 and associated uncertainty. Again, the new predicted trajectory and uncertainty are fed back as inputs to the predictive model module 135 so that predictions can be made regarding future states.

[0054] If the new predicted trajectory is acceptable, the user triggers the robotic system 140 to move the interventional device 146 according to the new control input. If the new predicted trajectory is not acceptable, the process of providing new untriggered control inputs, outputting a new predicted trajectory in response thereto, and displaying the new predicted trajectory overlaid on the current image is repeated until an acceptable predicted trajectory is obtained. At that point, a corresponding control input is triggered to move the interventional device 146 accordingly. This process is essentially continuous in that the new predicted trajectory is displayed substantially immediately following receipt of the new untriggered control input. Thus, for example, a user can see the effect of a change to a control input on the displayed predicted trajectory substantially simultaneously as the change to the control input is made.

[0055] 4 is a flow diagram illustrating a method for estimating and visualizing the trajectory of a robotically controlled interventional device on a display including a GUI, according to a representative embodiment, which may be performed by the system 100 described above under control of the processor 120 executing instructions stored, for example, as various software modules in memory 130.

[0056] 4, the method includes receiving a predictive model in block S411. For example, the predictive model may be a neural network model that has been previously trained for predicting trajectories of an interventional device and associated uncertainties. The training of the predictive model is based on training data from a previous image of the interventional device and corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the previous image. The training may be further based on training data from a current image of the interventional device and corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the current image.

[0057] In block S412, image data from at least one image of the anatomical structure is received. The at least one image is displayed on the display. The at least one image is a current image as described above. The at least one image shows a current position of the interventional device relative to the anatomical structure.

[0058] At least one untriggered control input is received in block S413 for controlling the robot to guide future movement of the interventional device from the current position. The at least one untriggered control input includes user input received through a user interface and status information received from the robotic system. The user input is provided by moving one or more interface devices such as a mouse, keyboard, trackball, joystick, microphone, video camera, touchpad, touch screen, etc. Because the at least one control input is untriggered, it does not immediately affect the movement of the robot or the interventional device controlled by the robot.

[0059] In block S414, the trajectory of the interventional device in at least one image is predicted by applying the received image data and at least one untriggered control input to the trained prediction model. That is, the trajectory of the interventional device in the image is predicted by providing the trained prediction model with the received image data (first input) of the anatomical structure and at least one untriggered control input (second input) corresponding to the received image data. The prediction model is composed of the imaging of the interventional device and the robot control parameters, and predicts the trajectory of the interventional device to guide the movement of the interventional device as shown in the image data of the first input based on the first and second inputs. In addition to the predicted trajectory, the trained prediction model also estimates an uncertainty associated with the predicted trajectory. Generally, the uncertainty increases the further the predicted trajectory extends from the current position of the interventional device. In a particular embodiment, the predicted trajectory and the associated uncertainty are returned to block S411 along with the corresponding untriggered control input, and the prediction model is further trained on the fly.

[0060] In block S415, the predicted trajectory of the interventional device is displayed on the display, overlaid on at least one image of the anatomical structure. In an embodiment, an estimated uncertainty associated with the predicted trajectory is also displayed overlaid on at least one image. For example, as described above, the predicted trajectory is displayed as a centerline on at least one image, and the associated uncertainty is displayed as lateral lines on either side of the centerline.

[0061] In block S416, it is determined whether the predicted trajectory has been identified as an acceptable trajectory for continuing to navigate the interventional device within the anatomical structure. This determination is made in response to an input by a user via a user interface to accept or reject the predicted trajectory. Alternatively or additionally, the acceptability of the predicted trajectory may be determined automatically. For example, as described above, the predicted trajectory may be determined automatically by using known image analysis techniques that provide a segmented predicted trajectory and comparing the segmented predicted trajectory to a desired trajectory or a predefined set of rules for navigating the interventional device.

[0062] If the predicted trajectory is not acceptable (S416: NO), the process returns to block S413, where a new untriggered control input for controlling the robot is received. The new untriggered control input is different from the previous untriggered control input to guide the future movement of the interventional device from the current position. Blocks S414-S416 are then repeated to identify an acceptable predicted trajectory.

[0063] If the predicted trajectory is acceptable (S416: YES), an untriggered control input is triggered to control the robot to guide movement of the interventional device according to the triggered control input in block 417. In an embodiment, block S417 may be substantially combined with block S416, and the predicted trajectory is determined to be acceptable in response to receiving a command to trigger the untriggered control input.

[0064] In block S418, the movement of the interventional device and / or robot in response to the triggered control input is displayed. The movement of the interventional device is monitored using the display to allow for continuous navigation of the interventional device through the anatomy. The actual movement of the interventional device can be displayed by collecting additional images of the interventional device in real time during the interventional procedure. Alternatively, the movement of the interventional device can be simulated by applying the triggered control input to a navigation model for navigating the interventional device using the same images as the reference images. The navigation model is essentially the same as the prediction model (e.g., neural network model) used for the prediction. In an embodiment, since the navigation model is essentially the same as the prediction model used for the prediction, the predicted trajectory is simply overlaid on the reference image. This overlay is rendered in a different color or shade, for example to distinguish it from the untriggered prediction of the subsequent frame. Also, an alert may be provided to the user if the uncertainty associated with the predicted trajectory is too high (e.g., above a predetermined threshold).

[0065] In block S419, it is determined whether navigation of the interventional device in the anatomical structure continues. This determination is made when untriggered control inputs continue to be received, for example, via a user interface, when a predefined target in the displayed image has not yet been reached, etc. If it is determined that navigation continues (block S419: yes), the process returns to block S413, where additional untriggered control inputs are received to control the robot to guide future movements of the interventional device. Blocks S414-S419 are then repeated to form a control loop for interactively guiding the interventional device using the robot. If it is determined that navigation continues (block S419: yes), the process returns. If it is determined that navigation is not continuing (block S419: no), the process ends.

[0066] 5 is a flow diagram of a method for determining and visualizing the trajectory of a robotically controlled interventional device on a display including a GUI, according to another representative embodiment. The method can be implemented by the system 100 described above, for example under the control of the processor 120 executing instructions stored in the memory 130. In the embodiment of FIG. 5, the trajectory of the interventional device is estimated without using a predictive model, thereby avoiding the training and application of a neural network.

[0067] 5, the method includes, in block S511, receiving previous image data from a previously acquired image (e.g., from previous image module 133) of the anatomical structure. The previous image shows the interventional device and the anatomical structure within the subject at a previous time. For example, the previous image shows the interventional device at time t=0, while the interventional device is currently at time t=n (n>0).

[0068] In block S512, previous position data of the interventional device is determined from the previous image data. The previous position data provides a previous position and trajectory of the interventional device in the previous image. The previous position data is determined, for example, by applying known segmentation techniques to the previous image. In particular, the trajectory of the interventional device can be determined by comparing the previous position of the interventional device in the previous image with a then-previous image of the interventional device, where the previous position describes how the interventional device is positioned in the context of the previous image and provides a spatial context for the interventional device. The previous trajectory describes how the device moves between multiple previous images and provides a temporal context for the interventional device.

[0069] In block S513, a control input is received via a control input module (e.g., control input module 132) for controlling the robot to guide the movement of the interventional device from the previous position. The control input includes user input and state information of the robotic system. The control input is a triggered control input that results in the actual movement of the interventional device.

[0070] In block S514, a displacement of the interventional device is calculated by applying the received control input to the previous position data from the previous image from a position of the interventional device at a previous position provided by the previous position data to a displaced position according to the control input.

[0071] In block S515, the calculated displacement of the interventional device is transformed into image coordinates using a previous image calibration of the previous image. For example, when tracking the interventional device using a technique that does not segment the interventional device in image space to provide position and / or trajectory information, calibration may be required because the tracking data is in a different coordinate space than the image data. However, the calibration information can be used in subsequent images.

[0072] In block S516, the current trajectory of the interventional device is displayed based on the image coordinates of the displacement. The current trajectory is displayed overlaid on the previous image being displayed. In this manner, no additional imaging is required during the interventional procedure, since the movement of the interventional device along the current trajectory is shown on the previous image.

[0073] 6 is a flow diagram of a method for controlling a robotically controlled interventional device by adjusting the visualized trajectory of the interventional device on a display including a GUI, according to another representative embodiment. The method is performed by the system 100 described above under control of the processor 120 executing instructions stored, for example, in memory 130. The embodiment shown in FIG. 6 is essentially the reverse of the embodiment shown in FIG. 4, in which the robot is automatically controlled by the robot controller in response to a user instruction to adjust the trajectory shown on the display.

[0074] 6, the method includes receiving a predictive model, such as a neural network model, at block S611. The predictive model is trained on determining control inputs for controlling a robot to guide an interventional device along a given predicted trajectory. The training may be based on training data from a previous image of the interventional device and corresponding control inputs to the robot to guide the movement of the interventional device as shown in the previous image. The training may further be based on training data from a current image of the interventional device and corresponding control inputs to the robot to guide the movement of the interventional device as shown in the current image.

[0075] 7 is a simplified block diagram illustrating an example implementation for training an encoder-decoder neural network architecture, according to a representative embodiment. The encoder-decoder neural network architecture may be similar to the neural network described above with reference to FIG. 3, where a current image is input to an encoder portion of the neural network, which generates a latent representation that is a reduced dimension of the input image. The latent representation is further used to estimate untriggered control inputs. The untriggered control inputs may be applied to the latent representation to generate a transformed latent representation. The transformed latent representation may be decoded to generate a predicted image, including a predicted trajectory of an interventional device, as described above.

[0076] 7, training images 711 are used as ground truth images to which the estimated control inputs are applied. During training of the neural network, predicted images 712 can be compared to, for example, training images 711. The predicted images 712 are images that represent the current position of the interventional device with the estimated control inputs.

number

number

[0077] During inference, when future images or control inputs are unknown, the desired future trajectory of the interventional device is specified by the user, for example, by drawing the desired trajectory in a GUI.

number

[0078] 8 is a simplified block diagram illustrating another implementation for training an encoder-decoder neural network architecture, according to a representative embodiment. With reference to FIG. 8, both a current image 812 and an image 811 containing a desired future trajectory of an interventional device (e.g., as a user drawn in a GUI) are input (e.g., as separate channels) to a neural network that directly estimates untriggered control inputs that transform the interventional device in the current image to the desired future trajectory. In this implementation, no fine-tuning is required during inference using the trained network. During training of the neural network, the estimated control inputs

number

[0079] 6, in block S612, current image data is received from a current image (e.g., from current image module 131) of an anatomical structure. The current image shows an anatomical structure within a subject, as acquired, for example, by imaging device 160, and a current position of an interventional device within the anatomical structure.

[0080] In block S613, the predicted trajectory of the interventional device is displayed overlaid on the current image. The predicted trajectory of the interventional device is predicted by applying the current image data and the current control inputs to a trained prediction model, as described above. In addition to the predicted trajectory, the trained prediction model can provide an estimated uncertainty associated with the predicted trajectory, which is also displayed.

[0081] In block S614, instructions are received from a user to adjust the displayed predicted trajectory to a desired position. For example, the GUI may include a touch screen that allows the user to drag the predicted trajectory to a desired position on the display screen. Or, as another example, the user may use a mouse to click and drag the predicted trajectory to a desired position.

[0082] In block S615, the control inputs corresponding to the desired position of the predicted trajectory are determined by applying the image coordinates of the predicted trajectory at the desired position to the predictive model, as described above.

[0083] In block S616, the determined control inputs are provided to a robot controller that, in response to the determined control inputs, automatically controls the robot to steer the interventional device along a desired predicted trajectory.

[0084] In various embodiments, the predictive model can be realized using different architectures of the encoder-decoder neural network model, for example incorporating convolutional layers and LSTM blocks. In general, the current image data is input to an encoder portion of the encoder-decoder neural network model, which generates a latent representation that is a reduced size of the input image. Untriggered control inputs are input to the encoder portion or applied to the latent representation to generate a transformed latent representation. The encoded image data and the (encoded) control inputs are decoded using the decoder portion of the encoder-decoder neural network model to generate a predicted image that includes a predicted trajectory of the interventional device.

[0085] In this context, Figure 9 is a simplified block diagram of an encoder-decoder neural network model configured to predict a device trajectory in a dimension-preserving architecture, according to a representative embodiment, where image data and control inputs are input to an encoder portion, and Figure 10 is a simplified block diagram of an encoder-decoder neural network model configured to predict a device trajectory in a dimension-varying architecture, according to a representative embodiment, where image data is input to an encoder portion and control inputs are input following encoding of the image data.

[0086] Referring to FIG. 9, the encoder-decoder neural network model 900 includes an encoder 910, a decoder 920, and a latent state 905 between the encoder 910 and the decoder 920. The encoder 910 includes LSTM blocks 911, 912, and 913, and the decoder 920 includes LSTM blocks 921, 922, and 923, all configured to capture temporal dependencies. Thus, the encoder-decoder neural network model 900 may be referred to as an LSTM neural network model. The encoder-decoder neural network model 900 is referred to as size-preserving because the feature layers after each of the LSTM blocks 911-913 and LSTM blocks 921-923 are the same size as the input data (as indicated by the matching block shapes). Of course, the functionality of the LSTM blocks 911-913 and 921-923 may be provided by other types of neural networks, such as RNNs, convolutional neural networks, CNNs, or transformers, without departing from the scope of the present teachings.

[0087] As shown in the figure, the encoder 910 receives as input both image data (ID) (e.g., previous image data) indicative of an interventional device in an anatomical structure and corresponding control inputs (CI) (e.g., previous and / or current control inputs) provided to the robot to guide the movement of the interventional device at the input level. That is, the image data and the corresponding control inputs are fused or combined at the earliest layer of the neural network model. A latent state 905 shows the hidden message after encoding of all the image data and the corresponding control inputs. The decoder 920 decodes the data in the latent state and transforms the encoded image data and control inputs to determine a predicted trajectory of the interventional device output from the encoder-decoder neural network model 900. The image data of each of the multiple images and the corresponding control inputs are input to the LSTM blocks 911-913 of the encoder 910. With respect to the LSTM blocks 921-923 of the decoder 920, the decoded data is input from the previous LSTM block in succession and output to the next LSTM block in succession.

[0088] Referring to FIG. 10, the encoder-decoder neural network model 1000 includes an encoder 1010, a decoder 1020, and a latent state 1005 between the encoder 1010 and the decoder 1020. The encoder-decoder neural network model 1000 also includes a fully connected layer 1030, an LSTM encoder 1040, and a combiner 1045. The encoder 1010 includes LSTM blocks 1011, 1012, and 1013, and the decoder 1020 includes LSTM blocks 1021, 1022, and 1023. Therefore, the encoder-decoder neural network model 1000 may be referred to as an LSTM neural network model. The encoder-decoder neural network model 1000 is referred to as variable-size because the size of the input data is gradually reduced in the encoder 1010 (as shown by the size of each of the LSTM blocks 1011-1013 narrowing from bottom to top) until the bottleneck layer is reached. Conversely, the dimensions of the latent state data are gradually increased in the decoder 1020 from the bottleneck layer back to the original input dimensions (as indicated by the dimensions of each of the LSTM blocks 1021-1023 expanding from bottom to top).

[0089] As shown, the encoder 1010 receives as input only image data (ID) (e.g., previous image data) that indicate the anatomical structure at the input level. The interventional device may also be visible in the image data. The latent state 1005 shows the hidden message after encoding of all image data by the encoder 1010. The control input (CI) (e.g., previous control input) corresponding to the image data is input to the fully connected layer 1030. The fully connected layer is configured to apply linear and nonlinear weights to the robot control input. In this case, the fully connected layer converts the robot control data into a representation that can be merged with the image data in the latent layer and corrects the dimensional mismatch between the image and the robot control data. The output of the fully connected layer 1030 is input to the LSTM encoder 1040. The LSTM encoder 1040 is configured to capture temporal dependencies similar to the encoder 1010 described above, but without the convolutional layer. A combiner 1045 combines the output of the LSTM encoder 1040 with the latent state data after the encoder 1010, and this combination is input to the decoder 1020. In another embodiment, the control input is combined with the latent state data at different stages of decoding by the decoder 1020, as indicated by the dashed arrow from the LSTM encoder 1040. This allows the encoder-decoder neural network model 1000 to learn dependencies between the robot control and image data at different scales and steps, resulting in more accurate predictions.

[0090] The decoder 1020 iteratively decodes the latent state data and transforms the combination of encoded image data and control inputs to determine a predicted trajectory of the interventional device output from the encoder-decoder neural network model 1000. Image data of each of the multiple images and the control inputs corresponding to the image data are input to LSTM blocks 1011-1013 of the encoder 1010. With respect to the LSTM blocks 1021-1023 of the decoder 1020, the decoded data is input consecutively from the previous LSTM block and output consecutively to the next LSTM block.

[0091] In the above embodiment, the predicted trajectory is determined for the future using current actual images (e.g., x-ray images, live fluoroscopic images) that are continuously collected during the interventional procedure. However, in an alternative embodiment, images may be collected discontinuously (e.g., periodically or intermittently at a low frame rate). In that case, navigation of the interventional device needs to be performed without actual images during the period between image collections. That is, when images are collected discontinuously, there are periods when the robot moves the interventional device within the anatomical structure without receiving new image data from the imaging device. To address this situation, in an alternative embodiment, a prediction of the interventional device's trajectory is provided, but the prediction is not a prediction of a future trajectory and / or position based on current image data, but a prediction of a current trajectory and / or position based on previous image data. In this case, the control input for controlling the robot to guide the interventional device is triggered and applied to move the interventional device up to the current time, rather than being triggered as in the previous embodiment.

[0092] 11 is a flow diagram illustrating a method for estimating and visualizing the trajectory of a robotically controlled interventional device using discontinuous imaging, according to a representative embodiment. The method can be performed by the system 100 described above under control of the processor 120 executing instructions stored, for example, as various software modules in the memory 130. In the embodiment of FIG. 11, the trajectory of the interventional device is estimated from the current image using a neural network model, and navigation of the interventional device continues through the robot without updating the image.

[0093] 11 , the method includes receiving a prediction model in block S1111. The prediction model has been previously trained for predicting the trajectory of the interventional device and associated uncertainty. The prediction model is, for example, a neural network model. The training may be based on training data from a previous image of the interventional device and corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the previous image. The training may further be based on training data from a current image of the interventional device and corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the current image.

[0094] In block S1112, current image data from a current image (e.g., from current image module 131) is received from a medical imaging device (e.g., imaging device 160). The current image is displayed on a display (e.g., display 124). The current image shows the interventional device and anatomical structures within the subject.

[0095] In block S1113, current position data of the interventional device is determined from the current image data. The current position data is determined, for example, by applying known image analysis / processing techniques to the current image. The current position data indicates how the interventional device is positioned in the context of the current image and provides a spatial context for the interventional device.

[0096] In block S1114, current control and state data for the robot is determined using robotics data from the robot system. The robotics data indicates the position and orientation of the robot while controlling the interventional device and includes, for example, kinematic data, joint information, encoder information, velocity, acceleration, end effector position, force, torque, and / or range and limits. The current control and state data for the robot can be correlated with the position data of the interventional device to determine the position of the robot when the interventional device is in its current position.

[0097] In block S1115, via a control input module (e.g., control input module 132), control inputs are received for controlling the robot to guide the movement of the interventional device from the current position. The control inputs include user inputs and state information of the robotic system. The control inputs are triggered control inputs that result in the actual movement of the interventional device and are intended to navigate the interventional device within the anatomical structure. The user inputs are provided by the user moving one or more user interfaces, such as a mouse, keyboard, trackball, joystick, microphone, video camera, touchpad, touch screen, etc.

[0098] In block S1116, the trajectory of the interventional device in the current image is predicted for the current position data of the interventional device by applying the current image data and the control inputs to the trained prediction model. The trained prediction model may also estimate the uncertainty associated with the predicted trajectory, as described above. In an embodiment, the prediction model may be further trained on the fly using the control inputs, the predicted trajectory, and the associated uncertainty.

[0099] In block S1117, the predicted trajectory of the interventional device is displayed on the display, overlaid on the current image. Because the current image has not been updated, it is assumed that the actual trajectory of the interventional device follows the displayed predicted trajectory, even though the user cannot see the actual location of the interventional device. The user can adjust user input in real time to change the course of the predicted trajectory as needed to maintain safe navigation of the interventional device relative to the current image being displayed. In an embodiment, an estimated uncertainty associated with the predicted trajectory is also displayed overlaid on the current image.

[0100] In block S1118, it is determined whether new current image data has been received from the medical imaging device from a newly acquired current image. The new current image shows the interventional device in a new (updated) position within the subject's anatomy. If new current image data has been received (block S1118: yes), the new current image is displayed on the display and the process returns to block S1113 to determine new current position data of the interventional device at the new position. Blocks S1114-S1118 are then repeated based on the new current position data. If new current image data has not been received (block S1118: no), the same current image of the interventional device and anatomy continues to be displayed and the process returns to block S1114 to determine updated current control and state data of the robot. Blocks S1115-S1118 are then repeated. The predicted trajectory of the interventional device may change to reflect any changes to the user input.

[0101] In accordance with various embodiments of the present disclosure, the methods described herein are implemented using a hardware computer system executing a software program stored on a non-transitory storage medium. Furthermore, in non-limiting exemplary embodiments, implementations such as distributed processing, component / object distributed processing, and parallel processing are possible. A virtual computer system process implements one or more of the methods or functions described herein. The processors described herein may also be used to support virtual processing environments.

[0102] Although estimating and visualizing a trajectory of a robotically controlled interventional device on a display has been described with reference to exemplary embodiments, it is understood that the terms used herein are terms of description and illustration, rather than terms of limitation. Changes may be made within the scope of the appended claims, as currently presented and as amended, without departing from the scope and spirit of the embodiments. Although estimating and visualizing a trajectory of a robotically controlled interventional device on a display has been described with reference to particular means, materials, and embodiments, the invention is not intended to be limited to the details disclosed. Rather, estimating and visualizing a trajectory of a robotically controlled interventional device on a display is intended to extend to all functionally equivalent structures, methods, and uses within the scope of the appended claims.

[0103] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of various embodiments. The illustrations do not completely describe all elements and features of the disclosure described herein. Many other embodiments will be apparent to those skilled in the art upon review of the present disclosure. Other embodiments can be utilized and derived from the present disclosure, such as structural and logical substitutions and changes can be made without departing from the scope of the present disclosure. Moreover, the illustrations are merely representative and may not be to scale. Certain proportions in the illustrations may be exaggerated and other proportions may be minimized. Thus, the present disclosure and the figures should be considered illustrative and not restrictive.

[0104] One or more embodiments of the present disclosure may be referred to herein, individually and / or collectively, by the term "invention", for convenience only and without any intention to spontaneously limit the scope of the present application to any particular invention or inventive concept. Also, although specific embodiments are illustrated and described herein, it should be understood that subsequent arrangements designed to achieve the same or similar purpose may be substituted for the specific embodiment shown. The present disclosure is intended to cover any and all subsequent adaptations or variations of the various embodiments. Combinations of the above embodiments with other embodiments not specifically described herein will be apparent to one of ordinary skill in the art upon reviewing the description.

[0105] The Abstract of the Disclosure is provided to comply with 37 CFR Rule 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above Detailed Description, various features may be grouped together or described in a single embodiment for purposes of streamlining the disclosure. This disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all features of any of the disclosed embodiments. Accordingly, the following claims are incorporated into the Detailed Description, with each claim standing on its own as separately defining the claimed subject matter.

[0106] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. Accordingly, the above disclosed subject matter should be considered as illustrative rather than restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent permitted by law, the scope of the present disclosure shall be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be limited or restricted by the foregoing detailed description.

Claims

1. 1. A system for estimating and visualizing the trajectory of an interventional device being robotically guided and inserted into an anatomy of a subject, comprising: At least one processor; a non-transitory memory for storing machine-executable instructions that, when executed by the at least one processor, cause the at least one processor to: receiving image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure; receiving at least one untriggered control input for controlling the robot to guide future movement of the interventional device from the current position; predicting at least one trajectory of the interventional device within the current image by providing the image data and the at least one untriggered control input to a model, the model being configured to predict a trajectory of the interventional device based on a first input of image data of the interventional device within the anatomical structure and a second input of a corresponding control input to the robot for guiding movement of the interventional device as indicated in the image data of the first input; providing a trigger command for triggering at least one untriggered control input to control the robot to guide movement of the interventional device according to the at least one triggered control input; A non-transient memory that allows A system comprising:

2. 2. The system of claim 1, wherein the instructions when executed further cause the at least one processor to automatically determine whether a predicted trajectory is acceptable, and wherein the trigger command is provided if the predicted trajectory is determined to be acceptable.

3. Automatically determining whether the predicted trajectory is acceptable includes: capturing a shape of the predicted trajectory to provide a segmented predicted trajectory; comparing the segmented predicted trajectory with a predetermined desired trajectory; determining that the predicted trajectory is acceptable if the segmented predicted trajectory substantially matches the predetermined desired trajectory; The system of claim 2 , comprising:

4. Automatically determining whether the predicted trajectory is acceptable includes: capturing a shape of the predicted trajectory to provide a segmented predicted trajectory; comparing the segmented predicted trajectory to a predetermined set of rules for navigating the interventional device within the anatomy; determining that the segmented predicted trajectory is acceptable if the segmented predicted trajectory substantially matches the predetermined rule set; The system of claim 2 , comprising:

5. The executed instructions further cause the at least one processor to: The system of claim 1 or 2, further comprising a user interface provided to enable a user to send the trigger command.

6. 6. The system of claim 5, wherein the user interface further provides the user with information regarding the at least one predicted trajectory to assist the user in determining whether the predicted trajectory is acceptable, thereby assisting the user to provide the trigger command.

7. The executed instructions further cause the at least one processor to: The system of any one of claims 1 to 6, further comprising: if the trigger command is not provided, providing at least one new untriggered control input for controlling the robot to alternatively guide the future movement of the interventional device from the current position.

8. The system of any one of claims 1 to 7, wherein the model is a trained model that predicts a trajectory of the interventional device based on training data from previous images of the interventional device within the anatomical structure and corresponding control inputs to the robot to guide movement of the interventional device as shown in the previous images.

9. The executed instructions further cause the at least one processor to:

9. The system of claim 1, further comprising: a controller configured to control a trajectory of the interventional device based on training data from at least one previous image of the interventional device within the anatomical structure, the received image data, at least one previous control input corresponding to the previous image, and the at least one untriggered control input to the robot for guiding movement of the interventional device as shown in the previous image.

10. The system of claim 1 , wherein the model is a convolutional long short-term memory (LSTM) neural network model.

11. 11. The system of claim 10, wherein the LSTM neural network model is configured with a dimension-preserving architecture to predict the at least one trajectory of the interventional device, and the first input of the image data and the second input of the corresponding control input are combined at an earliest layer of an encoder of the LSTM neural network model.

12. The system of claim 10, wherein the LSTM neural network model is configured with a scalable architecture to predict the at least one trajectory of the interventional device, and the first input of the image data is input at an earliest layer of an encoder of the LSTM neural network model, and the second input of the corresponding control input is input at or after a latent state between the encoder and decoder of the LSTM neural network model.

13. The system of claim 1 , wherein the model further predicts a trajectory of the interventional device based further on input of shape data of the interventional device shown in the image data of the first input.

14. The system of claim 1 , wherein the model further processes temporal sequences of imaging data such that, if the interventional device moves, the trajectory is predicted progressively over time.

15. 15. The system of claim 1, wherein the executed instructions further cause the at least one processor to use the model to estimate an uncertainty of the at least one predicted trajectory and to display the at least one predicted trajectory along with the estimated uncertainty overlaid on the current image of the anatomical structure.

16. The executed instructions further cause the at least one processor to: The system of any one of claims 1 to 15, further comprising: causing a user to overlay the at least one predicted trajectory of the interventional device on the current image of the anatomical structure to determine whether the at least one predicted trajectory is acceptable.

17. 17. The system of claim 15 or 16, wherein the executed instructions further cause the at least one processor to display the at least one predicted trajectory as a centerline, an outer line defining a margin along the centerline of the predicted trajectory, or the estimated uncertainty as color-coded pixels defining a margin along the centerline of the predicted trajectory.

18. The executed instructions further cause the at least one processor to: receiving additional image data from additional images of the anatomical structure after triggering the untriggered input; The system of any one of claims 1 to 17, further comprising, after triggering the untriggered control input, causing the actual trajectory of the interventional device to be displayed overlaid on the current image together with the predicted trajectory.

19. 19. The system of claim 1, further comprising a robot controller enabling control of the robot according to the at least one triggered control input.

20. 1. A system for estimating and visualizing the trajectory of a robotically guided interventional device inserted within an anatomy of a subject, comprising: At least one processor; a non-transitory memory for storing machine-executable instructions that, when executed by the at least one processor, cause the at least one processor to: receiving a model that predicts a trajectory of the interventional device based on data from images of the interventional device within the anatomical structure and corresponding control inputs to the robot to guide movement of the interventional device as shown in the images; receiving image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure; receiving an untriggered control input for controlling the robot to guide future movement of the interventional device from the current position; predicting a trajectory of the interventional device within the current image by applying the image data and the untriggered control inputs to the model; displaying the predicted trajectory of the interventional device overlaid on the current image of the anatomical structure for a user to determine whether the predicted trajectory is acceptable; receiving a trigger command to trigger the untriggered control input if the predicted trajectory is determined to be acceptable, and controlling the robot to guide movement of the interventional device according to the triggered control input; and if the predicted trajectory is determined to be unacceptable, receiving new untriggered control inputs for controlling the robot to alternatively guide the future movement of the interventional device from the current position. A system comprising:

21. 21. The system of claim 20, wherein the executed instructions further cause the at least one processor to train the model with respect to predicting a trajectory of the interventional device based on training data from previous images of the interventional device within the anatomical structure and corresponding control inputs to the robot to guide movement of the interventional device as shown in the previous images.

22. The system of claim 20 or 21, wherein the model is a neural network model trained on predicting the trajectory of the interventional device based on training data from previous images of the interventional device within the anatomical structure and corresponding control inputs to the robot to guide the movement of the interventional device as shown in those images.

23. The executed instructions further cause the at least one processor to: predicting alternative trajectories of the interventional device within the current image by applying the image data and the new untriggered control inputs to the model; causing the alternative predicted trajectory of the interventional device to be overlaid on the current image of the anatomical structure for the user to determine whether the alternative predicted trajectory is acceptable; The system of claim 20, further comprising: receiving the trigger command for triggering the new untriggered control input; and controlling the robot to guide movement of the interventional device in accordance with the triggered control input.

24. 24. The system of any one of claims 20 to 23, wherein the executed instructions further cause the at least one processor to automatically additionally determine the acceptability of the predicted trajectory by matching the predicted trajectory with a desired trajectory of the interventional device.

25. an imaging system that acquires the current image of the anatomical structure and provides the image data from the current image to the at least one processor; a robot controller that enables control of the robot according to the triggered control input; 25. The system of claim 20, further comprising:

26. 25. The system of claim 20, wherein the executed instructions further cause the at least one processor to use the model to estimate an uncertainty in the predicted trajectory and to overlay the predicted trajectory along with the estimated uncertainty on the current image of the anatomical structure.

27. 27. The system of claim 26, wherein the executed instructions further cause the at least one processor to display the predicted trajectory as a centerline and the estimated uncertainty as an outer line defining a margin along the centerline of the predicted trajectory.

28. 27. The system of claim 26, wherein the executed instructions further cause the at least one processor to display the predicted trajectory as a centerline and the estimated uncertainty as color-coded pixels defining a margin along the centerline of the predicted trajectory.

29. 27. The system of claim 21 or 26, wherein the training of the model comprises applying a dropout algorithm to estimate the uncertainty of the predicted trajectory.

30. The executed instructions further cause the at least one processor to: receiving additional image data from additional images of the anatomical structure after triggering the untriggered input; 30. The system of any one of claims 20 to 29, further comprising, after triggering the untriggered control input, causing the actual trajectory of the interventional device to be displayed overlaid on the current image together with the predicted trajectory.

31. The executed instructions further cause the at least one processor to: receiving additional image data from additional images of the anatomical structure while controlling the current movement of the interventional device; The system of any one of claims 20 to 30, further comprising: displaying an actual trajectory of the interventional device overlaid on the additional image while controlling the current movement of the interventional device.

32. 22. The system of claim 21, wherein the training of the model with respect to predicting the trajectory comprises unsupervised training.

33. 1. A method for estimating and visualizing the trajectory of a robotically guided interventional device inserted within an anatomy of a subject, comprising: training a neural network model for predicting a trajectory of the interventional device based on training data from a previous image of the interventional device and corresponding control inputs to the robot for guiding the movement of the interventional device as shown in the previous image; receiving image data from at least one image of the anatomical structure, the at least one image indicating a current position of the interventional device relative to the anatomical structure; receiving an untriggered control input for controlling the robot to guide future movement of the interventional device from the current position; predicting a trajectory of the interventional device in the at least one image by applying the image data and the untriggered control inputs to the trained neural network model; displaying the predicted trajectory of the interventional device overlaid on the at least one image of the anatomical structure; if the predicted trajectory is determined to be acceptable, triggering the untriggered control inputs to control the robot to guide movement of the interventional device according to the triggered control inputs; if the predicted trajectory is determined to be unacceptable, receiving new untriggered control inputs for controlling the robot to alternatively guide the future movement of the interventional device from the current position; A method comprising:

34. predicting alternative trajectories of the interventional device in the at least one image by applying the image data and the new untriggered control inputs to the trained neural network model; displaying the alternative predicted trajectories of the interventional device overlaid on the at least one image of the anatomical structure; triggering the new untriggered control input to control the robot to guide movement of the interventional device according to the triggered control input.

34. The method of claim 33, further comprising:

35. 34. The method of claim 33, wherein the predicted trajectory is displayed as a centerline and the estimated uncertainty of the predicted trajectory is displayed as an outer line defining a margin along the centerline of the predicted trajectory.

36. A computer program comprising instructions which, when executed on a processor, cause the processor to perform the steps of the method of claim 33.

37. 1. A system for estimating and visualizing the trajectory of a robotically guided interventional device inserted within an anatomy of a subject, comprising: At least one processor; a non-transitory memory for storing machine-executable instructions that, when executed by the at least one processor, cause the at least one processor to: receiving a model that predicts a trajectory of the interventional device based on data from images of the interventional device within the anatomical structure and corresponding control inputs to the robot to guide movement of the interventional device as shown in the images; receiving image data from a current image of the anatomical structure, the current image indicating a current position of the interventional device relative to the anatomical structure, the image data being collected on a discontinuous basis; determining current position data of the interventional device from the image data; determining current control and state data for the robot using robotics data from a robotic system for controlling the robot; receiving a triggered control input for controlling the robot to guide a current movement of the interventional device from the current position; predicting a trajectory of the interventional device within the current image by applying the image data and the triggered control input to the model; displaying the predicted trajectory of the interventional device overlaid on the current image of the anatomical structure to indicate an actual trajectory of the interventional device without receiving additional image data; determining whether new image data for a new current image has been received; a non-transitory memory configured to, when the new image data is received, determine new current position data of the interventional device from the new image data, determine new current control and state data of the robot using robotics data from a robot system for controlling the robot, iterate through the steps of: receiving triggered control inputs for controlling the robot to guide a current movement of the interventional device from the new current position data; and predicting a trajectory of the interventional device in the new current image by applying the new image data and the triggered control inputs to the model. A system comprising:

Images