Predictive motion mapping for flexible devices

Abstract

A controller (150) for an interventional medical device includes a memory (151) and a processor (152). The memory (151) stores instructions for execution by the processor (152). The instructions, when executed, cause the controller (150) to obtain at least one position of a distal end of the interventional medical device (101), identify motion at a proximal end of the interventional medical device (101), apply a first trained artificial intelligence to the motion at the proximal end of the interventional medical device (101) and the at least one position of the distal end of the interventional medical device (101), and predict motion along the interventional medical device (101) toward the distal end of the interventional medical device (101) during the interventional medical procedure. The controller (150) also obtains images of the distal end of the interventional medical device (101) from a medical imaging system (120) to determine when actual motion deviates from predicted motion.

Description

Background Technology

[0001] Using instruments on anatomical structures in minimally invasive surgery can be challenging when live medical imaging does not reveal unintended instrument movement. For some instruments, movement introduced proximally may not produce corresponding movement along the length of the instrument or at its distal end. Unintended movements along the length of the instrument include lateral translation and flexion. Variations in movement depend on the type of instrument, the patient's anatomy, and the curvature along the length of the instrument. Unintended instrument movement can also be due to instrument characteristics such as size, flexibility, torque transmission, and friction. The consequences of unintended movement can be adverse contact with tissue, such as vascular dissection or perforation. For example, movement introduced proximally into a long, thin instrument (e.g., a catheter or guidewire) can lead to unintended movement.

[0002] When performing minimally invasive surgery under the guidance of two-dimensional fluoroscopy, the three-dimensional (3D) motion of the instrument may be missed. Often, only the distal end of the instrument is within the fluoroscopic field of view (FOV), therefore any motion along the length of the instrument outside the FOV goes undetected. Additionally, due to fluoroscopic shortening in fluoroscopic imaging, distal motion may also often go undetected. Thus, large proximal movements of the instrument often do not elicit perceptible distal movements, but rather cause unintended behavior along the length of the instrument. A specific example of this problem occurs during peripheral vascular navigation in the leg. The catheter and guidewire are navigated from the femoral inlet, across the iliac angle, and down into the contralateral femoral artery. The X-ray images following the instrument do not show the iliac angle crossing, where the instrument may bend and return to the aorta.

[0003] Conventionally, given the start and end points of a tool procedure, the procedure can be plotted by simulating a microcatheter along its centerline and by assuming that the centerline consists of an alternating sequence of straight lines and curves. However, interference from external forces (e.g., physician manipulation of the catheter) is not considered.

[0004] The predictive motion mapping for flexible devices described in this paper addresses the problems mentioned above. Attached Figure Description

[0005] When with attachment Figure One The best understanding of the exemplary embodiments will be achieved by reading the following description. It should be emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and useful, the same reference numerals refer to the same elements.

[0006] Figure 1 The illustration depicts a system for predictive motion mapping of a flexible device according to a representative embodiment.

[0007] Figure 2 A method for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0008] Figure 3 Another method for mapping predictive motion of a flexible device is illustrated in accordance with another representative embodiment.

[0009] Figure 4A A hybrid process for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0010] Figure 4B Another hybrid process for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0011] Figure 5 A method for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0012] Figure 6A A hybrid process for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0013] Figure 6B Another hybrid process for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0014] Figure 6C A method for mapping predictive motion of a flexible device is illustrated in accordance with a representative embodiment.

[0015] Figure 7 A computer system on which a method for mapping predictive motion of a flexible device is implemented is illustrated in accordance with another representative embodiment. DETAILED DESCRIPTION

[0016] In the following detailed description, for the purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth to provide a thorough understanding of embodiments according to the present teachings. Descriptions of known systems, devices, materials, and methods can be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, such systems, devices, materials, and methods are within the scope of the present teachings and can be used in accordance with representative embodiments. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The defined terms are intended to encompass the technical meanings commonly understood and accepted by those skilled in the art of the present teachings in addition to the scientific technical meanings of the defined terms.

[0017] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the present inventive concept.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification and in the claims, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. Additionally, the terms "comprises," "comprising," "includes," "including" and / or "contains," "containing," when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0019] Unless otherwise stated, when an element or component is said to be "connected to," "coupled to" or "adjacent to" another element or component, it should be understood that the element or component can be directly connected or coupled to the other element or component, and that intervening elements or components can be present. That is, these terms and similar terms encompass instances where one or more intervening elements or components are present. However, when an element or component is said to be "directly connected" to another element or component, this only encompasses instances where the two elements or components are connected to each other without any intervening or intermediate elements or components.

[0020] The present disclosure is thus intended to cover all alternatives, modifications and equivalents falling within the scope of the present disclosure as defined by the claims. For the purposes of the present disclosure, the terms "exemplary", "by way of example", and "for example" are used herein to mean "an example of", not an "only example of". The phrase "by way of example" is used herein to introduce a list of one or more non-limiting examples. The phrase "for example" is used herein to introduce one or more examples of a procedure, feature, structure, or characteristic, in which the one or more examples are explained in further detail. Any example described herein as well as equivalents thereof can be employed in the practice of this disclosure.

[0021] As described herein, an expected range of motion at a distal end of an interventional medical device can be predicted from a particular motion of a proximal end of the interventional medical device. The expected motion along the length of the interventional medical device visible in a fluoroscopy FOV can be predicted, and an alert can be issued when the observed motion is outside the range of the expected motion. Additionally, based on the observed unexpected motion, a potential unplanned behavior outside the fluoroscopy FOV can be predicted. The warnings produced by the system can prevent additional damage to blood vessels and other undesirable outcomes. Predictive motion mapping of flexible devices can be used to track the motion of the interventional medical device and predict a coarse localization of the predicted unexpected behavior of the interventional medical device, even for portions of the interventional medical device that are outside the FOV of the medical imaging system used during the interventional medical procedure.

[0022] Figure 1 A system for predictive motion mapping of flexible devices is illustrated in accordance with representative embodiments.

[0023] In Figure 1 A control system 100 and an interventional medical device 101 are shown in

[0024] The control system 100 includes a medical imaging system 120, a motion detector 130, a workstation 140, a robot 160, and an artificial intelligence controller 180. The workstation 140 includes a controller 150, an interface 153, a monitor 155, and a touch panel 156. The controller 150 includes a memory 151 that stores instructions and a processor 152 that executes the instructions. The interface 153 interfaces the monitor 155 to the body of the workstation 140. The artificial intelligence controller 180 includes a memory 181 that stores instructions and a processor 182 that executes the instructions to implement one or more aspects of the methods described herein.

[0025] The characteristics of the interventional medical device 101 can influence how the interventional medical device 101 moves (both in terms of expected motion and unexpected and / or unplanned motion). For example, a soft guidewire can behave differently than a stiff guidewire (both in terms of expected motion and unexpected and / or unplanned motion). Thus, the characteristics of the interventional medical device 101 can be used as one or more of the bases to compensate for unexpected motion. Examples of interventional medical devices 101 include guidewires, catheters, microcatheters, and sheaths.

[0026] The medical imaging system 120 can be an interventional X-ray imaging system. The interventional X-ray imaging system can comprise an X-ray tube adapted to generate X-rays and an X-ray detector configured to acquire a time series of X-ray images such as fluoroscopy images. Examples of such X-ray imaging systems include a digital radiography-fluoroscopy system (e.g. Philips ProxiDiagnost), a stationary C-arm X-ray system (e.g. Philips Azurion), and a mobile C-arm X-ray system (e.g. Philips Veradius).

[0027] The medical imaging system 120 can be provided with an image processing controller configured to receive fluoroscopy images acquired during an interventional medical procedure and to output a segmentation result of the interventional device. The image processing controller can be implemented by / as the controller 150 shown, or can be implemented by / as another controller directly integrated with the medical imaging system 120. Figure 1 The image processing controller can be implemented by / as the controller 150 shown, or can be implemented by / as another controller directly integrated with the medical imaging system 120.

[0028] Segmentation of images produced by the medical imaging system 120 results in a representation of the surface of a structure such as an anatomical feature and the interventional medical device 101. The segmented representation can comprise, for example, a set of points in three-dimensional (3D) coordinates on the surface of the structure, and triangular facets defined by connecting adjacent groups of three points, such that the entire structure is covered by a mesh of non-intersecting triangular facets. A three-dimensional model of the interventional medical device 101 can be obtained by segmentation. Segmentation can also be represented as a binary mask, (x, y) coordinates of the interventional medical device 101 in image space, a two-dimensional spline or wireframe model. Segmentation can be computed by thresholding processes, template matching, active contour modeling, neural network based segmentation methods, and other segmentation methods. Segmentation can be provided for X-ray imagery generated by an X-ray imaging system or a three-dimensional ultrasound volume generated by an ultrasound imaging system.

[0029] The robot 160 can be used to control movement of the interventional medical device 101 under control of an operator. When the robot 160 controls the interventional medical device 101, motion at the proximal end of the interventional medical device 101 can be detected in accordance with motion of the robot 160.

[0030] The artificial intelligence controller 180 can comprise a plurality of controllers and can implement the first artificial intelligence and the second artificial intelligence as described herein. The artificial intelligence implemented by the artificial intelligence controller 180 can be produced by training in a dedicated training environment. The artificial intelligence controller 180 can be provided completely separate from other components of the control system 100 in Figure 1

[0031] ​The artificial intelligence controller 180 can be a neural network controller and can be used during an application phase during an interventional medical procedure. The artificial intelligence controller 180 is configured to receive motion information proximal to the interventional medical device 101. The artificial intelligence controller 180 is further configured to receive a fluoroscopy image and / or a segmented representation of the interventional medical device 101 in the fluoroscopy image from the medical imaging system 120. The artificial intelligence controller 180 can also receive a type of the interventional medical device 101, for example, from a drop-down menu provided via the monitor 155 or from an automatic detection of the interventional medical device 101. The interventional medical device can be automatically detected using object detection and classification from images of the interventional medical device 101 captured by an operating room camera prior to insertion of the interventional medical device 101 into the patient on the captured image. The artificial intelligence controller 180 can optionally operate based on constraints derived from the fluoroscopy image and / or the segmented representation of the interventional medical device 101. Constraints that can also be used as input by the artificial intelligence controller include a length of the interventional medical device 101, a maximum allowed curvature of the interventional medical device 101, and predicted and / or observed motion along the length of the interventional medical device 101 that is visible in the fluoroscopy FOV.

[0032] A result of the application of the first artificial intelligence and the second artificial intelligence by the artificial intelligence controller 180 can be a prediction of a rough positioning of where an unintended behavior, for example, a flexion, can occur outside of the fluoroscopy FOV based on an inconsistency between the predicted motion and the observed motion within the fluoroscopy FOV. Identifying an unintended behavior of the device motion within the fluoroscopy FOV helps to identify potential unplanned behavior that occurs outside of the fluoroscopy FOV. Another result of the application of the first artificial intelligence and the second artificial intelligence by the artificial intelligence controller 180 can be a generation of a warning when the predicted motion and the observed motion fall outside of a normal range of consistency. The warning generated using the trained artificial intelligence implemented by the artificial intelligence controller 180 can help to prevent the use of excessive force at the proximal end of the interventional medical device 101 when the expected motion is not observed at the distal end of the interventional medical device 101. This in turn will help to prevent adverse events, for example, vessel dissection or perforation, false aneurysm, vessel rupture, and other undesirable outcomes, for example, accidental displacement of the damaged portion, breakage of the guide wire, etc.

[0033] Although not shown, the artificial intelligence controller 180 can also be configured to receive a type of the interventional medical device 101, for example, from a drop-down menu provided via the monitor 155 or from an automatic detection of the interventional medical device 101. The interventional medical device can be automatically detected using object detection and classification from images of the interventional medical device 101 captured by an operating room camera prior to insertion of the interventional medical device 101 into the patient on the captured image. Figure 1The control system 100 in the system 200 can also include a feedback controller to alert the physician when unintended behavior outside the FOV is predicted. The feedback controller can produce a warning, e.g., an alarm sound, a printed message on a fluorescent display, haptic feedback to the proximal end of the interventional medical device 101, or robotic control guidance. The robotic control guidance can be provided with a displayed direction to modify the motion at the proximal end of the interventional medical device 101. The displayed direction can include, for example, a suggestion to move the robotic control forward, backward, or laterally, a suggestion regarding touchscreen or joystick motion for a Corindus CorPath, a suggestion regarding knob rotation for a steerable sheath, a steerable guide catheter, or a transesophageal echocardiography (TEE) probe. The feedback controller can also provide robotic control guidance in a closed loop system, e.g., by sending commands to an autonomous robot to automatically pull back. The feedback controller can also provide robotic control guidance to the collaborative control robot to suppress further forward motion if forward motion can lead to buckling.

[0034] While the control system 100 is primarily described in the context of an X-ray imaging system, the control system 100 can also be included or incorporated into interventional ultrasound imaging systems as well as fixed and mobile interventional X-ray imaging systems. The control system 100 can be used for various fluoroscopy-based interventional medical procedures, including but not limited to interventional vascular procedures.

[0035] Figure 2 A method for predictive motion mapping of a flexible device is illustrated in accordance with a representative embodiment.

[0036] At S210, Figure 2 The method of the system 200 starts with training artificial intelligences. The trained artificial intelligences can include a first artificial intelligence and a second artificial intelligence, which are trained using different inputs to produce different outputs. In addition, the output from the first artificial intelligence can be an input to the second artificial intelligence. Furthermore, a first prediction or inference from the first artificial intelligence can be output from the first artificial intelligence and input to the second artificial intelligence, and the second artificial intelligence can output a second prediction or inference based on using the first prediction or inference from the first artificial intelligence as an input. Realistic situation information of a coarse localization of unintended behavior of an interventional medical device can be used to train the artificial intelligences, and the trained artificial intelligences can be used once the artificial intelligences are deployed. The characteristics of the first artificial intelligence and the second artificial intelligence are explained more in the following paragraphs. The training of the artificial intelligences at S210 can be performed entirely before the deployment of the artificial intelligences. In one embodiment, the adaptive artificial intelligences can use post-deployment feedback to self-improve via reinforcement learning or other learning methods.

[0037] The output from Figure 1The artificial intelligence controller 180 can implement the training of the first trained artificial intelligence and the second trained artificial intelligence. For example, in a plurality of training episodes for a plurality of interventional medical devices, the artificial intelligence controller 180 can input at least one position of a distal end of an interventional medical device, detect a motion at a proximal end of the interventional medical device, and detect a motion along the interventional medical device towards the distal end of the interventional medical device caused by the motion at the proximal end of the interventional medical device. In the plurality of training episodes for the plurality of interventional medical devices, the artificial intelligence controller 180 can additionally be configured to input a type of the interventional medical device, a type of an anatomical structure or a type of a procedure, or other contextual information that varies for different interventional medical procedures. The position of the distal end of the interventional medical device in the training can be obtained from images, e.g. by deriving from medical images from a medical imaging system. The plurality of training episodes can further comprise predicting a predicted motion along the interventional medical device towards the distal end of the interventional medical device based on the at least one position of the distal end of the interventional medical device and the detected motion at the proximal end of the interventional medical device. The plurality of training episodes can further comprise detecting an actual motion along the interventional medical device towards the distal end of the interventional medical device and determining a loss based on a difference between the predicted motion and the actual motion. The first trained artificial intelligence can establish a relationship between the motion at the proximal end of the interventional medical device and the motion along the interventional medical device towards the distal end of the interventional medical device, and the first artificial intelligence can be updated based on each loss determined from a difference between the predicted motion and the detected actual motion.

[0038] After the training at S210, the artificial intelligence can be provided for use. Then, Figure 1 The artificial intelligence controller 180 in the interventional medical system can implement the first artificial intelligence.

[0039] For an interventional medical procedure, the first artificial intelligence can additionally be trained for different types of interventional medical devices 101. The first trained artificial intelligence can optionally input at least one of the type of interventional medical device 101, the type of interventional medical procedure, an anatomical landmark, or at least one physical property of the patient and act based on these items. A drop-down menu can be provided to the clinician to select the type of interventional medical device 101. The predicted motion along the interventional medical device 101 towards the distal end of the interventional medical device 101 can additionally be predicted based on the selected type of interventional medical device. Similarly, the prediction of the predicted motion can additionally be based on the anatomy of the patient in the interventional medical procedure, the position of the medical imaging system, or the physical properties of the interventional medical device 101. Alternatively, the type of interventional medical device 101 subject to the first artificial intelligence can be automatically selected by performing object detection and classification prior to the interventional medical procedure, e.g., prior to the insertion of the interventional medical device 101 into the patient. The detection and classification can be implemented based on images captured by an operating room camera. Alternatively, the detection and classification can be implemented using a model trained by machine learning to detect and classify different types of interventional medical devices. The training dataset of machine learning for creating the model can include training instances comprising X-ray images of a plurality of different interventional medical devices. The training data can only contain normal or expected motion at the distal end of the plurality of different interventional medical devices, such that the artificial intelligence will learn to predict the normal motion at the distal end of different interventional medical devices and, during inference, if the subsequently observed motion is not similar to the predicted normal motion, a warning or alarm can be generated and issued. The training data can be collected using shape sensing technology such as FORS. FORS provides 3D shape information along the length of the device, thus allowing to confirm that the data contains expected motion and does not contain unexpected motion such as flexion.

[0040] In operation during the interventional medical procedure, the first artificial intelligence can also be implemented based on additional contextual information, e.g., the target region or anatomy, segmentation of the surrounding anatomy, pose of the C-arm, and the target region, to allow the first artificial intelligence to learn when to expect a perspective shortening. In addition, the first artificial intelligence can receive constraints on the output, e.g., the length of the interventional medical device 101 in the fluoroscopy image, or the maximum allowed curvature of the interventional medical device 101.

[0041] The first artificial intelligence can be a neural network, e.g., a convolutional neural network, an encoder-decoder network, a generative adversarial network, a capsule network, a regression network, a reinforcement learning agent, and can use the motion information at the proximal end of the interventional medical device 101 and the fluoroscopy image at the initial time t to predict the motion or motion field along the length of the interventional medical device 101 visible in the fluoroscopy field of view (FOV). The motion or motion field observed between the fluoroscopy images at time t and time t+n can be compared to the motion predicted by the first artificial intelligence to learn the expected range of motion that can be observed in the fluoroscopy FOV. This time t+n can be after a particular motion at the proximal end is completed, or be another arbitrary time (e.g., when a particular motion at the proximal end is occurring). The predicted motion is compared to the observed (ground truth) motion by computing a loss function, e.g., mean squared error, or mean absolute error, or Huber loss, or any loss that computes the difference between two motion vectors (R 2 , R 3 ). The motion can be represented by vector parameterization and / or non-vector parameterization and / or motion field. The parameterization can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angular axes that represent rotation and / or translation (e.g., including direction and magnitude of translation and rotation)

[0042] At S220, Figure 2 The method of S220 includes identifying motion at the proximal end of the interventional medical device. The motion in the proximal end can include forward translation, lateral translation, and rotation along an axis. The motion in the proximal end is the motion induced by the user or robot, and can include any motion induced by the user or robot that controls the interventional medical device 101. The motion information at the proximal end of the interventional device can be obtained from a sensing device. Examples of sensing devices that can capture and provide such motion information include a device tracker, an inertial measurement unit (IMU) sensor, a monocular or stereo camera system that acts as an optical tracking system, a linear encoder, a torque encoder, or an optical encoder. Examples of device trackers include an optical tracking sensing system, an optical tracking system, an electromagnetic tracking system, or an optical shape sensing mechanism. Examples of IMU sensors include sensors that measure angular velocity, force, and possibly magnetic field, which have components such as accelerometers, gyroscopes, and possibly magnetometers. Examples of linear encoders include optical linear encoders, magnetic linear encoders, and capacitive inductive linear encoders.

[0043] At S225, Figure 2The method of includes obtaining a medical image of the interventional medical device. The medical image obtained at S225 can be obtained by the medical imaging system 120. The medical image obtained at S225 can be used to obtain at least one position of the distal end of the interventional medical device 101 from an image of the distal end of the interventional medical device 101. The medical image can be a fluoroscopic image of a portion of the interventional medical device within a field of view of the medical imaging system. The medical image obtained at S225 is a portion of the interventional medical device that is distal facing. Based on the medical image obtained at S225, the method of includes obtaining a first artificial intelligence trained to identify a motion at a proximal end of the interventional medical device and to predict a motion along the interventional medical device distal to the proximal end of the interventional medical device based on the identified motion at the proximal end of the interventional medical device and an image of the interventional medical device distal to the proximal end of the interventional medical device. The first artificial intelligence is trained to identify a motion at a proximal end of the interventional medical device and to predict a motion along the interventional medical device distal to the proximal end of the interventional medical device based on the identified motion at the proximal end of the interventional medical device and an image of the interventional medical device distal to the proximal end of the interventional medical device. Figure 2 Embodiments of include obtaining an image of the distal end of the interventional medical device prior to S230, for example when the segmented representation of the interventional medical device is used as input to the first artificial intelligence applied at S230. In embodiments of Figure 4B In (discussed later), the image(s) of the distal end are represented as a fluoroscopic frame f t The medical image can include a single or time series of fluoroscopic images containing the distal end of the interventional medical device 101. The medical image can be automatically segmented by the image processing controller using methods such as thresholding, template matching, active contour modeling, multi-scale ridge enhancement filter, or deep learning based segmentation algorithms.

[0044] At S230, Figure 2 The method of includes applying the trained first artificial intelligence to the identified motion at the proximal end of the interventional medical device and the medical image of the distal end of the interventional medical device when the motion is applied to the proximal end of the interventional medical device. The trained first artificial intelligence is an artificial intelligence trained to find a correlation between a motion applied at the proximal end of the interventional medical device and a motion received at the distal end of the interventional medical device observed in the interventional medical image.

[0045] At S240, the first artificial intelligence predicts a motion along the interventional medical device distal to the proximal end based on the motion identified at S220 at the proximal end of the interventional medical device 101 and the image of the interventional medical device distal to the proximal end at S225. The first artificial intelligence can be implemented by receiving a fluoroscopic image of the segmented representation of the interventional medical device at S225 without unintended / unplanned behavior (e.g., flexion) that initially covers the image of the interventional medical device 101. Since the first artificial intelligence has been trained prior to the interventional medical procedure, the first artificial intelligence can use the initial information from the segmented representation of the interventional medical device as a basis to determine a normal or expected proximal to distal motion mapping.

[0046] At S250, Figure 2 The method of includes obtaining an image of the interventional medical device from the medical imaging system. The image of the interventional medical device obtained at S250 can be an image of the distal end of the interventional medical device and / or an image distal to the interventional medical device.

[0047] At S255, Figure 2 The method of S255 comprises segmenting the interventional medical device in the image from the medical imaging system. The segmentation yields a segmented representation of the interventional medical device. S255 can be optional and can also be performed on the image of the interventional medical device distal end on towards obtained at S225 and inputted into the trained first artificial intelligence at S230.

[0048] At S257, actual motion is detected from the image from the medical imaging system.

[0049] At S260, the detected actual motion of the interventional medical device is compared to the predicted motion of the interventional medical device. In Figure 2 The predicted motion of the interventional medical device compared at S260 in S255 is the motion predicted at S240.

[0050] At S270, it is determined whether the actual motion deviates from the predicted motion. Such deviation can be identified according to a binary classification process or can be based on one or more threshold values, a scoring algorithm or other process that determines whether the actual motion of the interventional medical device is within an expected range from the predicted motion.

[0051] If the actual motion does not deviate from the predicted motion (S270 = No), no alert is generated. If the actual motion deviates from the predicted motion (S270 = Yes), an alert is generated at S280.

[0052] Additionally, at S271, Figure 2 The method of S271 comprises predicting a coarse localization of motion along the interventional medical device outside of a field of view from the image of the interventional medical device. Additionally, the second artificial intelligence can predict a predicted confidence of the predicted coarse localization. The coarse localization predicted at S271 can be predicted by the second artificial intelligence described herein. The second artificial intelligence can be implemented by the artificial intelligence controller 180 and implements a localization neural network. The second artificial intelligence is configured to receive the predicted device motion and the device motion observed from the fluoroscopy image. Data for training the second artificial intelligence to predict a coarse localization of motion occurring outside of the FOV of the medical imaging system can be obtained using shape sensing technology such as FORS.

[0053] The second artificial intelligence can be implemented by a trained neural network, e.g., a convolutional neural network, an encoder-decoder network, a generative adversarial network, a capsule network, a regression network, a reinforcement learning agent. The second artificial intelligence can use the predicted motion and the observed motion at the distal end as input to predict whether and where an unexpected / unplanned behavior (e.g., flexion) appears outside the fluoroscopy FOV and compare its prediction to the ground truth information from the ground truth localization at S210 obtained in the training, e.g., from the FORS. The predicted localization is compared to the ground truth localization from the ground truth information by computing a loss function such as mean squared error or mean absolute error or Huber loss. The second artificial intelligence can generate a warning if an unexpected / unplanned behavior is predicted. The warning can be generated based on the presence or absence of the unplanned behavior predicted in the manner described herein.

[0054] At S281, a display is generated for the predicted coarse localization along the interventional medical device that is outside the FOV of the image.

[0055] Figure 3 Another method for predictive motion mapping of a flexible device is illustrated according to another representative embodiment.

[0056] At S310, Figure 3 The method of S310 includes inputting a detected motion at a proximal end of the interventional medical device.

[0057] At S320, at least one position of a distal end of the interventional medical device is detected.

[0058] At S330, a first artificial intelligence is trained to predict a motion along the interventional medical device towards the distal end.

[0059] At S360, prior to applying the motion at the proximal end of the interventional medical device, a motion along the interventional medical device towards the distal end is predicted based on the motion at the proximal end and a medical image of the distal end of the interventional medical device.

[0060] At S370, an actual motion along the interventional medical device towards the distal end is detected. The actual motion is detected from a medical image or a segmented representation of a portion of the interventional medical device that is within a field of view of a medical imaging system.

[0061] At S380, Figure 3 The method of S380 includes determining a loss function based on a difference between the predicted motion and the actual motion towards the distal end of the interventional medical device.

[0062] At S385, the first neural network is updated based on the determined loss function and the process returns to S330.

[0063] In Figure 3 embodiments, when training the first neural network, the first neural network can be updated at S385. In embodiments, if the data generated during operation represents normal or expected proximal-to-distal motion mapping and can be reliably used as ground truth, the data can be used to update the first neural network at S385 after the first neural network is run.

[0064] Figure 4A FIG. illustrates a hybrid process for predictive motion mapping of a flexible device, according to representative embodiments.

[0065] In Figure 4A embodiments, the first artificial intelligence 410A and the second artificial intelligence 415A are trained using a first loss function (loss function #1) and a second loss function (loss function #2) based on inputs including motion at the proximal end and the distal end of the interventional medical device. As explained below, the inputs to the first artificial intelligence 410A and the second artificial intelligence 415A during training are explained by way of example for corresponding features of Figure 4B

[0066] Figure 4B FIG. illustrates another hybrid process for predictive motion mapping of a flexible device, according to representative embodiments.

[0067] In Figure 4B ​A proximal-to-distal predictive motion mapping process and system is schematically represented in FIG. 1. An interventional medical device 401 (e.g., a guidewire) and a medical imaging system 420 (e.g., a fluoroscopic X-ray medical imaging system) are used to train artificial intelligence, for example, in a controlled environment. The interventional medical device 401 can include a first region that is visible in medical imaging and a second region that is not visible in medical imaging. The first region and the second region can change in operation as the FOV of the medical imaging system 420 changes. During operation, the first region and the second region can change as the view of the medical imaging changes. A first artificial intelligence can be trained to establish a relationship between motion at the proximal end of the interventional medical device 401 and resulting motion at the distal end of the interventional medical device 401 by using motion at the proximal end of the interventional medical device 101 and medical images of the distal end of the interventional medical device 101 as motion is applied to the proximal end of the interventional medical device 101 and predicting resulting motion at the distal end, then comparing the resulting motion to observed motion at the distal end. A second artificial intelligence can be trained to predict a coarse localization of an unplanned behavior of the interventional medical device 401 by using observed motion and predicted motion at the distal end and comparing the predicted coarse localization of the unplanned behavior to a ground truth coarse localization of the unplanned behavior. Training of the second neural network 415B can use ground truth information of the coarse localization of the distal end of the interventional medical device so that once the second neural network 415B is deployed, learning from the training can be used.

[0068] In Figure 4B the segmentation of the interventional medical device 401 in the fluoroscopic frame f t 420 is provided as input to the first neural network 410B along with motion applied at the proximal end of the interventional medical device 401. Of course, Figure 4B the segmentation of the interventional medical device 401 in FIG. 1 is used for a first region that includes a portion of the interventional medical device 401 that is in the field of view (FOV) of the medical imaging system. Another portion of the interventional medical device 401 is not within the field of view (FOV) of the medical imaging system. The first neural network 410B outputs a point-wise motion estimate along the length of the segmentation representation of the interventional medical device 401. The point-wise motion estimate is compared to an observation that is computed from the segmentation representation of the interventional medical device 401 at a later fluoroscopic frame f t+n . In other words, Figure 4B the first neural network 410B in FIG. 1 learns how motion at the proximal end of the interventional medical device 401 causes a correlation of motion at points along the distal end of the interventional medical device 401.

[0069] Also in Figure 4BIn particular, the predicted point-wise motion estimate along the length of the segmented representation of the interventional medical device and the actual motion observed from the segmented representation are input to a second neural network 415B. Of course, the input to the second neural network 415B is for the estimated motion and the actual motion of the interventional medical device 401 in a first region, which includes the portion of the interventional medical device 401 in the field of view (FOV) of the medical imaging system. The output of the second neural network 415B is a prediction of a coarse localization of an unplanned motion in a second region, which includes the portion of the interventional medical device 401 outside the field of view of the medical imaging system. The estimated coarse localization in the second region is compared to a ground truth localization in the second region obtained from a mechanism such as a shape sensing technique. For example, the ground truth localization can be obtained via a shape sensing technique (e.g., Fiber Optic RealShape (FORS) from Philips).

[0070] As mentioned above, in Figure 4B In particular, the first neural network 410B is trained to output a point-wise motion estimate along the length of the interventional medical device 401 towards the distal end based on the motion at the proximal end of the interventional medical device 401 as input and the segmented representation of the interventional medical device 401. The training of the first neural network 410B is based on feedback of a first loss function (loss function 1) that reflects the difference between the predicted motion and the observed motion at the distal end of the interventional medical device 401. The second neural network 415B is trained to output a coarse localization of an unplanned behavior of the interventional medical device 401 based on the point-wise motion estimate and the actual observed motion within the field of view of the medical imaging system. The training of the second neural network 415B is based on feedback of a second loss function (loss function 2) that reflects the difference between the output of the coarse localization and the ground truth localization of the unplanned behavior. In Figure 5 During the training of the second neural network, the predicted coarse localization of the unplanned behavior can be confirmed via the second loss function using actual localization from, for example, optical shape sensing. Once the second neural network is trained to an acceptable accuracy, the use of the predicted localization during training can be applied during operation even without the use of actual localization. As a result, the unplanned behavior of the interventional medical device 401 can be predicted and coarsely localized.

[0071] Figure 5 A method for predictive motion mapping of a flexible device is illustrated in accordance with a representative embodiment.

[0072] The training of the first and second artificial intelligences is described in Figure 6A In particular, the first neural network 410B is trained to output a point-wise motion estimate along the length of the interventional medical device 401 towards the distal end based on the motion at the proximal end of the interventional medical device 401 as input and the segmented representation of the interventional medical device 401. The training of the first neural network 410B is based on feedback of a first loss function (loss function 1) that reflects the difference between the predicted motion and the observed motion at the distal end of the interventional medical device 401. The second neural network 415B is trained to output a coarse localization of an unplanned behavior of the interventional medical device 401 based on the point-wise motion estimate and the actual observed motion within the field of view of the medical imaging system. The training of the second neural network 415B is based on feedback of a second loss function (loss function 2) that reflects the difference between the output of the coarse localization and the ground truth localization of the unplanned behavior. In

[0073] At S520, the segmentation of the distal end of the interventional medical device is input to the first neural network as a second input.

[0074] At S525, the first neural network is applied to the inputs from S510 and S520.

[0075] At S530, the predicted motion of the distal end of the interventional medical device is output from the first neural network.

[0076] At S540, the observed motion of the distal end of the interventional medical device is compared to the predicted motion of the distal end of the interventional medical device to produce a first loss function.

[0077] At S545, the first neural network is updated. The process of training the first neural network continues by returning to S525 until the process ends.

[0078] At S550, the predicted motion of the distal end of the interventional medical device from the output of the first neural network is input to the second neural network as a first input.

[0079] At S560, the observed motion of the distal end of the interventional medical device is input to the second neural network as a second input.

[0080] At S565, the second neural network is applied.

[0081] At S570, the second neural network outputs a coarse localization of an unplanned behavior outside the field of view of the imaging device.

[0082] At S580, the ground truth localization of the unplanned behavior of the interventional medical device is compared to the predicted coarse localization of the unplanned behavior to produce a second loss function.

[0083] At S585, the second loss function is fed back to update the second neural network. After S585, the process of training the second neural network returns to S565.

[0084] Figure 6A FIG. 1 illustrates a hybrid process for predictive motion mapping of a flexible device, according to a representative embodiment.

[0085] In Figure 6B corresponding features of the representative embodiments described herein. The inputs to the first artificial intelligence 610A and the second artificial intelligence 615A during operation are explained by way of example of corresponding features of the representative embodiments described below. Figure 6B

[0086] Figure 6B ​Another hybrid process for predictive motion mapping of a flexible device is illustrated in accordance with a representative embodiment.

[0087] In Figure 6B another proximal-to-distal predictive motion mapping process and system is schematically represented. The hybrid process and system in Figure 6B may be used during an interventional medical procedure. As in the hybrid process of Figure 6C , for any fluoroscopy frame f t , the segmentation of the interventional medical device 601 in a first region within the field of view of the medical imaging is provided along with the motion imposed at the proximal end of the interventional medical device 601 as input to a first neural network 610B. The first neural network 610B outputs a point-wise motion estimate along the length of the segmented representation of the interventional medical device 601. The estimate from the first neural network 610B is compared to an observation computed from the segmented representation of the interventional medical device 601 in the first region within the field of view of the medical imaging system at a later fluoroscopy frame f t+n . When these two motions (estimated and observed) do not match, these two motions become input to a second neural network 616B which predicts a coarse localization of where an unplanned motion in a second region outside the fluoroscopy FOV is likely to be.

[0088] Figure 6C A method for predictive motion mapping of a flexible device is illustrated in accordance with a representative embodiment.

[0089] In Figure 7 , at S610, the observed motion at the proximal end of the interventional medical device is input to a first neural network as a first input.

[0090] At S620, the segmentation of the interventional medical device at the distal end of the interventional medical device is input to the first neural network as a second input. The segmented representation can be provided as a binary mask of the segmented representation of the interventional medical device in a fluoroscopy image.

[0091] At S625, the first neural network is applied to the first input at S610 and the second input at S620.

[0092] At S630, a predicted motion at the distal end of the interventional medical device is output from the first neural network as a predicted motion. The first neural network can be a trained encoder-decoder network. The predicted motion can be a motion along the length of the interventional medical device towards the distal end and can be output by the trained encoder-decoder network.

[0093] At S650, the predicted motion at the distal end of the interventional medical device output from the first neural network is input to the second neural network as a first input.

[0094] At S660, the observed motion at the distal end of the interventional medical device is input to the second neural network as a second input. The second neural network can be a trained convolutional neural network.

[0095] At S665, the second neural network is applied to the predicted motion at S650 and the observed motion at S660.

[0096] At S670, the second neural network outputs a coarse localization of an unintended / unplanned behavior outside the FOV of the medical imaging system. The second neural network can localize the unintended / unplanned behavior in the interventional medical device 101 outside the fluoroscopy FOV based on the inconsistency between the predicted motion and the observed motion at the distal end of the interventional medical device within the FOV of the medical imaging system. As described above, this prediction can be used to generate an alert.

[0097] As described in the above embodiments, one or more deep learning algorithms are trained to learn a relationship or mapping between the motion applied at the proximal end of the interventional medical device and the observed motion at the distal end of the interventional medical device. The captured input motion can include manual motion or mechanical motion, e.g., robotic motion or robotic-assisted motion, and can be rotational and / or translational. In alternative embodiments, the deep learning algorithm(s) can also learn proximal-to-distal mapping of velocity fields at multiple points, acceleration, inertia, spatial configuration, tangential (angular) motion, and linear velocity or acceleration. The learning by the deep learning algorithm(s) can take into account specific parameters of the interventional medical device. During the procedure, given the motion applied to the interventional medical device at the proximal end and medical images of the distal end of the interventional medical device as the motion is applied to the proximal end of the interventional medical device, the control system 100 estimates the motion of the interventional medical device at the distal end. The control system 100 can also learn to correlate the difference between the predicted device motion and the observed device motion at the distal end and different unobserved device behavior that emerges outside the fluoroscopy FOV. The control system 100 can then alert the physician of possible unplanned behavior in the interventional medical device outside the FOV of the medical imaging system, thus preventing possible vascular damage or other undesirable outcomes.

[0098] In one embodiment, consistent with the teachings above, a deep learning model can be trained to predict the motion at the distal end of the interventional medical device 101 from the two-dimensional coordinates of the segmented representation of the interventional medical device 101 or from the two-dimensional coordinates of a spline fit to the segmented representation of the interventional medical device in the fluoroscopy view.

[0099] In another embodiment, a recurrent neural network (RNN) architecture (e.g., a long short-term memory (LSTM) network, a temporal convolutional network (TCN), or a transformer) can be used to observe the segmentation representation of the interventional medical device 101 in multiple fluoroscopy frames (t_0 to t_n) in order to better inform the motion prediction in frame t_n

[0100] In another embodiment, a deep learning model can be trained to predict the position(s) of the segmentation representation of the interventional medical device in frame t_(n+1) from the segmentation of the interventional medical device in frame t_n only or in frames t_0 to t_n. This prediction can be directly compared to the device observed in fluoroscopy frame t_(n+1).

[0101] In another embodiment, a known three-dimensional model or a set of parameters, rules, or characteristics of the interventional medical device 101 is used to inform the prediction of the motion or velocity of the interventional medical device 101.

[0102] In another embodiment, a deep learning algorithm can be trained to learn a proximal-to-distal mapping of device-specific parameters. Examples of device-specific parameters include velocity field, acceleration, inertia, spatial configuration, tangential / angular velocity, and linear or acceleration velocity at multiple points. In this embodiment, the predicted parameters can be compared to the measured parameters.

[0103] In another embodiment, a machine learning algorithm can use the predicted motion information and the observed motion information to classify the observation as normal or abnormal. Examples of machine learning algorithms include a one-class support vector machine (SVM) classifier or a deep learning-based classifier. In this embodiment, a warning can be generated when an abnormality is detected.

[0104] In another embodiment, a deep learning network can be trained to predict the motion at the distal end of the interventional medical device from an ultrasound image of the distal end of the interventional medical device. The input to the deep learning network can be provided from the ultrasound image, a binary mask of the segmentation representation of the interventional medical device in the ultrasound, two-dimensional (x, y) coordinates of the segmentation representation of the interventional medical device 101 in the ultrasound, or spline-fitted two-dimensional (x, y) coordinates of the segmentation representation of the interventional medical device 101 in the ultrasound.

[0105] In yet another embodiment, the deep learning network can be trained to additionally learn the confidence of the predicted motion at the distal end of the interventional medical device based on any other method that agrees with the ground truth or determines the confidence or uncertainty during training. The control system 100 can learn the type of input motion at the proximal end of the interventional medical device 101 that produces a trustworthy estimate of the motion at the distal end or the type of fluoroscopy view that is associated with trustworthy motion prediction. For example, the control system 100 can learn that displaying a perspective shortened view can not produce a very trustworthy estimate of the motion at the distal end. The confidence of the predicted motion at the distal end predicted by the first deep learning network can additionally be input into a second network to predict a coarse localization of the unplanned behavior along the interventional medical device 101 outside the field of view of the medical imaging system. Similarly, the second deep learning network can be trained to additionally predict the confidence of the predicted coarse localization of the unplanned behavior along the interventional medical device 101 outside the field of view of the medical imaging system.

[0106] In yet another embodiment, the intravascular robotic system measures the force being applied at the tip of the catheter to display the measurement of the force on the console or incorporate the measurement of the force into the control loop. This feature can alert the clinician that the persistent force is dangerous, thus reducing the likelihood of perforation or other damage to the vessel wall. In this embodiment, the detection of abnormal or unplanned device behavior (e.g., buckling) is incorporated into the safety mechanism or control loop of the robotic system. For example, if buckling is predicted to occur at the end effector (distal portion), the risk of vessel wall perforation is high, and the robotic actuators will slow down or trigger an emergency stop. The user can be informed on the console of the corrective action and asked to perform the corrective action. Alternatively, in the case of a semi- or fully-automated robot, the control system 100 can automatically retract the end effector, turn it to a different orientation, and approach the cannulation again. If the buckling occurs in the medical portion of the guidewire, the control system 100 will adjust the settings of the controller 150 (e.g., PID controller parameters, including gains and motor speeds) in the background using preprogrammed rules or sophisticated predictive models. Only if the controller adjustment fails or is ineffective, the user is informed, thus avoiding operator overload, cognitive burden, and reduced trust in the robotic system. The robotic system can also learn the operator’s preferences and only inform the user if the buckling occurs in certain regions, with a certain intensity or frequency, or for certain durations.

[0107] In yet another embodiment, at the proximal end of the interventional medical device 101, complex input in the form of a touch screen or joystick manipulation is introduced in order to control the Corindus CorPath; or complex input in the form of a knob rotation is introduced in order to control a steerable sheath, a steerable guide catheter, a TEE probe. If the control system 100 detects an unplanned behavior in the interventional medical device, the control system 100 can suggest a mechanism for eliminating the unplanned behavior in the environment of the input device. Examples of input devices include a touch screen, a joystick, and a knob.

[0108] Figure 7 A computer system according to another representative embodiment is illustrated, on which the method for predictive motion mapping of a flexible device is implemented.

[0109] Figure 7 The computer system 700 shows a complete set of components for a communication device or a computer device. However, a "controller" as described herein can be implemented with fewer than all of the components of the computer system 700. For example, by a combination of memory and processor. The computer system 700 can include some or all of the elements of one or more component means of the systems for predictive motion mapping of a flexible device herein, but any such means can not necessarily include one or more of the elements described for the computer system 700 and can include other elements not described. Figure 7

[0110] With reference to the Figure 1 , the computer system 700 includes a set of software instructions that can be executed to cause the computer system 700 to perform any of the methods or computer-based functions disclosed herein. The computer system 700 can operate as a standalone device or can be connected, e.g., using a network 701, to other computer systems or peripheral devices. In embodiments, the computer system 700 performs logical processing based on digital signals received via an analog-to-digital converter.

[0111] In a networked deployment, the computer system 700 operates in the capacity of a server or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. The computer system 700 can also be implemented as or incorporated into various devices, such as a personal computer (PC), a tablet PC, a set-top box (STB), a Figure 7 ​The controller 150, fixed computer, mobile computer, personal computer (PC), laptop computer, tablet computer, or other machine capable of (sequentially or otherwise) executing a set of software instructions specifying actions to be performed by that machine. The computer system 700 can be incorporated as a device or incorporated into a device, which in turn is included in an integrated system including additional devices. In embodiments, the computer system 700 can be implemented using electronic devices that provide voice, video, or data communications. Furthermore, although the computer system 700 is illustrated as a single system, the term "system" should also be considered to include any collection of systems or subsystems that individually or jointly execute one or more sets of software instructions to perform one or more computer functions.

[0112] like Figure 1 As shown, the computer system 700 includes a processor 710. The processor 710 can be considered as... Figure 1 The processor 152 of the controller 150 executes instructions to implement some or all aspects of the methods and processes described herein. The processor 710 is tangible and non-transient. As used herein, the term "non-transient" should not be interpreted as a perpetual state characteristic, but rather as a characteristic of a state that will persist for a period of time. The term "non-transient" specifically denies transient characteristics, such as carrier waves or signals or other forms of characteristics that exist only momentarily at any time and place. The processor 710 is an article of manufacture and / or a machine part. The processor 710 is configured to execute software instructions to perform the functions described in the various embodiments herein. The processor 710 may be a general-purpose processor or a part of an application-specific integrated circuit (ASIC). The processor 710 may also be a microprocessor, microcomputer, processor chip, controller, microcontroller, digital signal processor (DSP), state machine, or programmable logic device. The processor 710 may also be a logic circuit (including a programmable gate array (PGA) such as a field-programmable gate array (FPGA)) or another type of circuit comprising discrete gate and / or transistor logic units. The processor 710 may be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or some combination thereof. Furthermore, any processor described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in or coupled to a single device or multiple devices.

[0113] The term“processor” as used herein encompasses an electronic component which is able to execute a program or machine executable instruction. References to the computing device comprising“a processor” should be interpreted as including more than one processor or processing core, as in a multi-core processor. A processor can also refer to a collection of processors or processing cores located on a single computing device, e.g. according to a parallel computing, grid computing or grid / parallel computing grid arrangement. The term computing device should also be interpreted to include a collection or network of computing devices each including a processor.

[0114] The computer system 700 also includes a main memory 720 and a static memory 730 that either or both communicate with the processor 710 via the bus 708. The main memory 720 and the static memory 730 can be considered as machine storage media that can be non-transitory. The main memory 720 and the static memory 730 can be considered as machine storage media that can be non-transitory. Figure 7 The memory 151 of the controller 150 in the system 100 is a representative example of the memory 720 and the static memory 730 in the computer system 700, and stores instructions to implement some or all aspects of the methods and processes described herein. The memory described herein is a tangible storage medium for storing data and executable software instructions that is non-transitory during the time software is stored in the memory. The term“non-transitory” as used herein does not The main memory 720 and the static memory 730 are articles of manufacture and / or machine components. The main memory 720 and the static memory 730 are computer readable media from which a computer (e.g., the processor 710) can read data and executable software instructions. Each of the main memory 720 and the static memory 730 can be implemented as one or more of: a random access memory (RAM), a read only memory (ROM), a flash memory, an electrically programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), registers, a hard disk, a removable disk, a tape, a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a floppy disk, a Blu-ray disk, or any other form of storage medium known in the art. The storage medium can be volatile or non-volatile, secure and / or encrypted, unsecure and / or unencrypted.

[0115] “Memory” is an example of a computer-readable storage medium. Computer memory is any memory that a processor can directly access. Examples of computer memory include, but are not limited to, RAM, registers, and register files. The reference to “computer memory” or “memory” should be interpreted as potentially referring to multiple memories. Memory can be, for example, multiple memories within the same computer system. Memory can also be multiple memories distributed across multiple computer systems or computing devices.

[0116] As shown in the figure, the computer system 700 may also include, for example, a video display unit 750 (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT)). Additionally, the computer system 700 includes an input device 760 (e.g., a keyboard / virtual keyboard, a touch-sensitive input screen, or a voice input unit with voice recognition) and a cursor control device 770 (e.g., a mouse, a touch-sensitive input screen, or a pad). The computer system 700 may also optionally include a disk drive unit 780, a signal generation device 790 (e.g., a speaker or a remote control), and a network interface device 740.

[0117] In an embodiment, such as ​ As shown, the disk drive unit 780 includes a computer-readable medium 782 in which one or more sets of software instructions 784 (software) are embedded. The set of software instructions 784, to be executed by the processor 710, is read from the computer-readable medium 782. Additionally, the software instructions 784, when executed by the processor 710, perform one or more steps of the methods and processes described herein. In embodiments, the software instructions 784 reside wholly or partially in main memory 720, static memory 730, and / or reside wholly or partially in the processor 710 during operation by the computer system 700. Furthermore, the computer-readable medium 782 may include the software instructions 784 or receive and execute the software instructions 784 in response to a propagated signal, causing a device connected to the network 701 to transmit voice, video, or data on the network 701. The software instructions 784 may be sent or received on the network 701 via a network interface device 740.

[0118] In embodiments, special-purpose hardware implementations, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components, are constructed to implement one or more of the methods described herein. One or more embodiments described herein can implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that enable the modules to operate as described. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. Nothing in this application should be interpreted as a limitation on a method, system, or device being implemented only in software except as expressly set forth in the claims.

[0119] In accordance with various embodiments of the present disclosure, the methods described herein can be implemented using a hardware computer system that runs software programs. Additionally, in example non-limiting embodiments, implementations can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing can implement one or more of the methods or functions described herein, and can use processors described herein to support virtual processing environments.

[0120] Accordingly, predictive motion mapping of a flexible device provides guidance about the actual positioning of an interventional medical device, for example, when the interventional medical device is used in an interventional medical procedure under the guidance of live two-dimensional fluoroscopy imaging. Characteristics of the interventional medical device that can cause unplanned motion can be used as a basis to compensate for the unplanned motion. Similarly, anatomical structures of the patient that can cause unplanned motion can be used as a basis to compensate for the unplanned motion.

[0121] The present disclosure describes a system that learns the range of motion or behavior that can be expected at a distal end given a particular motion or action at a proximal end and a current position or configuration of the distal end of an interventional medical device. The system can then predict the expected motion along the length of a guidewire that is visible in fluoroscopy and alert when observed motion exceeds the expected range of motion. Additionally, the system can observe the type of unplanned motion in the fluoroscopy field of view (FOV) and predict where unexpected or unplanned behavior outside the fluoroscopy FOV occurs (e.g., close to the FOV or far from the FOV).

[0122] Nonetheless, predictive motion mapping of a flexible device is not limited to application of the specific details described herein, but is applicable to additional embodiments in which one or more inputs to the first artificial intelligence and the second artificial intelligence are different than the specific details described for the embodiments herein.

[0123] While predictive motion mapping for flexible devices has been described with reference to several exemplary embodiments, it will be understood that the words which have been used are words of description and illustration, rather than the words of limitation. Changes (e.g., to the claimed subject matter and to the exemplary embodiments) can be made within the scope and spirit of the aspects of predictive motion mapping for flexible devices as described above, as reference to the claims and as modifications that are made will be understood and appreciated by those skilled in the art. Although predictive motion mapping for flexible devices has been described with reference to particular means, materials and embodiments, the predictive motion mapping for flexible devices is not intended to be limited to the particulars disclosed; rather, the predictive motion mapping for flexible devices extends to all functionally equivalent structures, methods and uses (e.g., those within the scope of the appended claims).

[0124] The descriptions of embodiments described herein are intended to provide an overall understanding of the structure of various embodiments. The descriptions are not intended to be a complete description of all elements and features of the disclosure described herein. Many other embodiments will be apparent to those skilled in the art upon reviewing the present disclosure. Other embodiments can be utilized and derived therefrom, such that structural and logical substitutions and changes can be made without departing from the scope of the disclosure. Additionally, the drawings are merely representative and can not be drawn to scale. Certain proportions of the illustrations can be exaggerated, while other proportions can be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative in nature and not a limitation.

[0125] In this document, the terms "invention" and "embodiments" can be used to refer to one or more embodiments of the disclosure, individually and / or collectively, only for convenience and without intending to limit the scope of this patent or patent disclosure. Furthermore, while specific embodiments have been illustrated and described, it will be appreciated that any subsequent arrangement designed to achieve the same or similar purpose, is considered within the scope of the disclosure. The disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of reasonable skill in the art upon reviewing the description.

[0126] This abstract of disclosure is provided in accordance with 37 C. FR § 1.72(b) and is to be understood at the time of filing not to be used for interpreting or limiting the scope or meaning of the claims. Additionally, in the foregoing detailed description, various features may be grouped together or described in a single embodiment for the purpose of simplification. This disclosure should not be construed as reflecting an intention that the claimed embodiments require more features than expressly recited in each claim. Rather, as reflected in the claims, the inventive subject matter may refer to all features of fewer than any of the disclosed embodiments. Therefore, the claims are incorporated into the detailed description, with each claim independently defining a separately claimed subject matter.

[0127] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in this disclosure. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the claims are intended to cover all such modifications, enhancements, and other embodiments falling within the true spirit and scope of this disclosure. Accordingly, to the fullest extent permitted by law, the scope of this disclosure will be determined by the broadest permissible interpretation of the claims and their equivalents, and should not be limited by the foregoing detailed description.

Claims

1. A controller (150) for an interventional medical device, comprising: Memory (151), its storage instructions, and A processor (152) that executes the instructions, wherein the instructions, when executed by the processor (152), cause the controller (150) to perform the following operations: At least one location distal to the interventional medical device (101) is obtained during the interventional medical procedure; Movement at the proximal end of the interventional medical device (101) is identified during the interventional medical procedure; The first trained artificial intelligence is applied to the movement at the proximal end of the interventional medical device (101) and at least one location at the distal end of the interventional medical device (101); Based on the application of the first trained artificial intelligence to the movement at the proximal end of the interventional medical device (101) and at least one position at the distal end of the interventional medical device (101), a predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) during the interventional medical procedure is predicted. After the movement at the proximal end of the interventional medical device (101) is identified, an image of the distal end of the interventional medical device (101) is obtained from the medical imaging system (120); Based on the image from the interventional medical device (101), the actual movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) is compared with the predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101), and Determine when the actual motion deviates from the predicted motion.

2. The controller (150) according to claim 1, wherein, The at least one location of the distal end of the interventional medical device (101) is obtained based on an image of the distal end from the medical imaging system (120); and For multiple locations along the interventional medical device (101) toward the distal end of the interventional medical device (101), the predicted motion is predicted along the interventional medical device (101) toward the distal end of the interventional medical device (101).

3. The controller (150) according to claim 1, in, The first trained artificial intelligence is configured to take into account at least one of the following: the type of the interventional medical device (101), the type of the interventional medical procedure, anatomical landmarks, or at least one physical characteristic of the patient. The predicted motion is predicted along the interventional medical device (101) toward the distal end of the interventional medical device (101) based on the input to the first trained artificial intelligence.

4. The controller (150) according to claim 1, wherein, The instructions, when executed by the processor (152), also cause the controller (150) to perform the following operations: An alarm is generated when the actual motion deviates from the predicted motion.

5. The controller (150) according to claim 1, wherein, The instructions, when executed by the processor (152), also cause the controller (150) to perform the following operations: A rough localization based on the actual motion and the predicted motion, and by applying second-trained artificial intelligence to predict unplanned behavior along the interventional medical device (101) outside the field of view of the medical imaging system (120).

6. A system for controlling an interventional medical device, comprising: A motion detector (130) is configured to detect motion at the proximal end of the interventional medical device (101); as well as A controller (150) includes a memory (151) for storing instructions and a processor (152) for executing the instructions, wherein the instructions, when executed by the processor (152), cause the system to perform the following operations: At least one location distal to the interventional medical device (101) is obtained during the interventional medical procedure; The movement at the proximal end of the interventional medical device (101) is identified during the interventional medical procedure; The first trained artificial intelligence is applied to the movement at the proximal end of the interventional medical device (101) and at least one location at the distal end of the interventional medical device (101); Based on the application of the first trained artificial intelligence to the movement at the proximal end of the interventional medical device (101) and at least one position at the distal end of the interventional medical device (101), a predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) during the interventional medical procedure is predicted. After the controller (150) detects the movement at the proximal end of the interventional medical device (101), it obtains an image of the distal end of the interventional medical device (101) from the medical imaging system (120); Based on the image from the interventional medical device (101), the actual movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) is compared with the predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101), and Determine when the actual motion deviates from the predicted motion.

7. The system according to claim 6, further comprising: The medical imaging system (120), The at least one location of the distal end of the interventional medical device (101) is obtained based on an image of the distal end of the interventional medical device (101) from the medical imaging system (120); and The images from the medical imaging system (120) are segmented to identify the interventional medical device (101) imaged by the medical imaging system (120).

8. The system according to claim 6, further comprising: An artificial intelligence controller (180) implements the first trained artificial intelligence and the second trained artificial intelligence, wherein the instructions, when executed by the processor (152), cause the system to further perform the following operations: In multiple training sessions for multiple interventional medical devices, at least one position of the distal end of the interventional medical device is input, movement at the proximal end of the interventional medical device is identified, and actual movement along the interventional medical device toward the distal end of the interventional medical device caused by the movement at the proximal end of the interventional medical device is detected. Based on the at least one location of the distal end of the interventional medical device and the identified movement at the proximal end of the interventional medical device, a predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device is predicted. The loss is determined based on the difference between the predicted motion and the actual motion; The artificial intelligence trained in the first stage establishes a relationship between the movement at the proximal end of the interventional medical device and the movement along the interventional medical device toward the distal end of the interventional medical device. The first trained artificial intelligence is updated based on each loss determined according to the difference between the predicted motion and the actual motion.

9. The system according to claim 6, wherein, The instruction causes the system to perform the following further operations: In multiple training phases, the approximate location of the interventional medical device outside the field of view of the medical imaging system (120) is input as real-world information. Based on the predicted motion and the actual motion, and by applying a second artificial intelligence (415A), a coarse localization of the interventional medical device outside the field of view of the medical imaging system (120) is predicted; and a loss is determined based on the difference between the actual localization of the interventional medical device and the predicted coarse localization of the interventional medical device outside the field of view of the medical imaging system (120); and The second AI (415A) is updated based on each loss.

10. The system according to claim 6, further comprising: A robot (160) that controls the movement of the proximal portion of the interventional medical device (101); as well as An interface (153) is used to output an alarm based on the predicted motion along the interventional medical device (101) toward the distal end of the interventional medical device (101) within the field of view of the medical imaging system (120).

11. A computer program product for controlling an interventional medical device, the computer program product comprising computer program units that, when run by a processor, cause the processor to perform a method comprising the following operations: At least one location of the distal end of the interventional medical device (101) is obtained from the medical imaging system (120) during the interventional medical procedure; Movement at the proximal end of the interventional medical device (101) is identified during the interventional medical procedure; The first trained artificial intelligence is applied to the movement at the proximal end of the interventional medical device (101) and at least one location at the distal end of the interventional medical device (101); Based on the application of the first trained artificial intelligence to the movement at the proximal end of the interventional medical device (101) and at least one position at the distal end of the interventional medical device (101), a predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) during the interventional medical procedure is predicted. After the movement at the proximal end of the interventional medical device (101) is identified, an image of the distal end of the interventional medical device (101) is obtained from the medical imaging system (120); Based on the image from the interventional medical device (101), the actual movement along the interventional medical device (101) toward the distal end of the interventional medical device (101) is compared with the predicted movement along the interventional medical device (101) toward the distal end of the interventional medical device (101), and Determine when the actual motion deviates from the predicted motion.

12. The computer program product according to claim 11, wherein, The method further includes: The image of the interventional medical device (101) is segmented to identify the interventional medical device (101) as imaged by the medical imaging system (120).

13. The computer program product according to claim 11, wherein, The method further includes: Predict the predicted motion and the confidence level of the predicted motion; and Based on the actual motion, the predicted motion, and the confidence level of the predicted motion, and by applying second-trained artificial intelligence to predict the predicted coarse location of unplanned behavior along the interventional medical device (101) outside the field of view of the medical imaging system (120) and the confidence level of the predicted coarse location.

14. The computer program product according to claim 11, wherein, The method further includes: The predicted motion is predicted based on at least one of the following: the type of interventional medical device, the patient's anatomy during the interventional medical procedure, the location of the medical imaging system (120), or the physical characteristics of the interventional medical device (101); and An alarm is output when the actual motion deviates from the predicted motion.

15. The computer program product according to claim 11, wherein, The method further includes: Based on the actual motion and the predicted motion, and by applying second-trained artificial intelligence to predict the coarse localization of unplanned behavior along the interventional medical device (101) outside the field of view of the medical imaging system (120); and Generate a rough localization of the predicted unplanned behavior along the interventional medical device (101) outside the field of view of the medical imaging system (120).

Images