Predictive Motion Mapping for Flexible Devices

Abstract

The controller 150 for the interventional medical device includes a memory 151 and a processor 152. The memory 151 stores instructions for execution by the processor 152. When executed, the instructions cause the controller 150 to obtain at least one position of the distal end of the interventional medical device 101, identify motion at the proximal end of the interventional medical device 101, apply a first trained artificial intelligence to the motion at the proximal end of the interventional medical device 101 and the at least one position of the distal end of the interventional medical device 101, and predict motion along the interventional medical device 101 toward the distal end of the interventional medical device 101 during the interventional medical procedure. The controller 150 also obtains images of the distal end of the interventional medical device 101 from the medical imaging system 120 to determine when the actual motion deviates from the predicted motion.

Description

Background Art

[0001]

[0001] Using an instrument with an anatomical structure in minimally invasive procedures can be difficult when the instrument's unexpected movements cannot be seen by live medical imaging. For some instruments, when an action is introduced at the proximal end, corresponding actions along the length of the instrument and corresponding actions at the distal end of the instrument may not occur. Unexpected actions along the length of the instrument include lateral movement and buckling. The variations in actions depend on the type of instrument, the patient's anatomical structure, and the curvature along the length of the instrument. The unexpected actions of the instrument can also be due to instrument characteristics such as dimensions, flexibility, torque transmission, and friction. As a result of unintended actions, incorrect contact with tissue, such as cutting or perforating a blood vessel, can occur. As an example, unexpected actions are caused by actions introduced at the proximal end of a long, thin instrument such as a catheter or a guide wire.

Summary of the Invention

[0002]

[0002] When a minimally invasive procedure is performed under the guidance of two-dimensional (2D) fluoroscopy, the aspect of the three-dimensional (3D) movement of the instrument may be overlooked. In many cases, only the distal end of the instrument is within the fluoroscopy field of view (FOV), so the actions transmitted along the length of the instrument outside the FOV are not perceived. Furthermore, in many cases, the actions at the distal end are also not perceived due to foreshortening of the fluoroscopy imaging. Therefore, even if there is a large action at the proximal end of the instrument, as a result, the action at the distal end is often not perceived, leading to unexpected behavior along the length of the instrument. A specific example of this problem occurs when performing peripheral vascular navigation of the leg. The catheter and guide wire access from the thigh, cross the iliac angle, and move down the contralateral femoral artery. The X-ray image following the instrument cannot capture the crossing of the iliac angle where the instrument can buckle back into the aorta.

[0003]

[0003] Conventionally, when the start of an instrument course and the end of the instrument course are given, a microcatheter having a microcatheter tube along the centerline of the microcatheter is modeled, and it is assumed that the centerline of the microcatheter is composed of a line in which a straight line and a curve are alternately continuous, and thus the course is plotted. However, interference from external forces such as the operation of the catheter by a doctor is not considered.

[0004]

[0004] The predictive motion mapping for the flexible device described herein addresses the above concerns.

[0005]

[0005] Exemplary embodiments are best understood from the following detailed description when read in conjunction with the figures of the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily enlarged or reduced for clarity of discussion. Where applicable and practical, like reference numerals refer to like elements.

Brief Description of the Drawings

[0006]

Figure 1

[0006] FIG. shows a system for predictive motion mapping of a flexible device according to an exemplary embodiment.

Figure 2

[0007] FIG. shows a method for predictive motion mapping of a flexible device according to an exemplary embodiment.

Figure 3

[0008] FIG. shows another method for predictive motion mapping of a flexible device according to another exemplary embodiment.

Figure 4A

[0009] FIG. shows a hybrid process for predictive motion mapping of a flexible device according to an exemplary embodiment.

Figure 4B

[0010] FIG. shows another hybrid process for predictive motion mapping of a flexible device according to an exemplary embodiment.

Figure 5

[0011] FIG. showing a method for predictive operation mapping of a flexible device according to an exemplary embodiment.

Figure 6A

[0012] FIG. showing a hybrid process for predictive operation mapping of a flexible device according to an exemplary embodiment.

Figure 6B

[0013] FIG. showing another hybrid process for predictive operation mapping of a flexible device according to an exemplary embodiment.

Figure 6C

[0014] FIG. showing a method for predictive operation mapping of a flexible device according to an exemplary embodiment.

Figure 7

[0015] FIG. showing a computer system in which a method for predictive operation mapping of a flexible device according to another exemplary embodiment is implemented.

DETAILED DESCRIPTION OF THE INVENTION

[0007]

[0016] In the following detailed description of the invention, for the purpose of explanation and not limitation, representative embodiments that disclose specific details are described to provide a complete understanding of the embodiments according to the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted so as not to obscure the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the knowledge of those skilled in the art are within the scope of the present teachings and can be used in accordance with the representative embodiments. It should be understood that the technical terms used herein are for the purpose of describing specific embodiments only and are not intended to be limiting. The defined terms are additionally defined to the technical and scientific meanings of the generally understood and accepted defined terms in the technical field of the present teachings.

[0008]

[0017] In this specification, terms such as first, second, third, etc. are used to describe various elements or components, but it will be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another. Thus, the first element or component considered below could also be referred to as the second element or component without departing from the teachings of the concepts of the present invention.

[0009]

[0018] The technical terms used in this specification are for the sole purpose of describing specific embodiments and are not intended to be limiting. When used in this specification and the appended claims, the singular form is intended to include both the singular and plural forms unless the context clearly dictates otherwise. Further, the terms "comprising", "having", and / or similar terms, when used in this specification, specify the presence of the described features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0010]

[0019] Unless otherwise noted, when an element or component is said to be "connected", "coupled", or "adjacent" to another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these terms and similar terms encompass cases where one or more intermediate elements or components are employed to connect two elements or components. However, when an element or component is said to be "directly connected" to another element or component, it encompasses only cases where the two elements or components are connected to each other without any intervening or intermediate elements or components.

[0011]

[0020] The present disclosure is intended to disclose one or more of the advantages specifically mentioned below through one or more of its various aspects, embodiments, and / or specific features or sub-components. For the purpose of illustration and without the intention of limitation, exemplary embodiments that disclose specific details are described to provide a complete understanding of the embodiments according to the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein are within the scope of the appended claims. Further, to avoid obscuring the description of the exemplary embodiments, the description of well-known devices and methods may be omitted. Such methods and devices are within the scope of the present disclosure.

[0012]

[0021] As described herein, the expected motion range at the distal end of an interventional medical device is predicted from a specific motion at the proximal end of the interventional medical device. An expected motion transmitted in the longitudinal direction of the interventional medical device visible in the fluoroscopy FOV is predicted, and an alert is issued when the observed motion is outside the range of the expected motion. Further, based on the observed unexpected motion, an unexpected behavior that may occur outside the fluoroscopy FOV is predicted. The warnings generated by this system prevent accidental damage to blood vessels and other undesirable results. The predicted motion mapping of the flexible device tracks the motion of the interventional medical device and predicts a predicted approximate estimation position for roughly estimating the unexpected behavior of the interventional medical device even for a part of the interventional medical device that is not within the FOV of the medical imaging system used during the interventional medical procedure.

[0013]

[0022] FIG. 1 shows a system for predicted motion mapping of a flexible device according to an exemplary embodiment.

[0014]

[0023] In FIG. 1, control system 100 is shown together with an invasive medical device 101. The invasive medical device 101 has a proximal end P closest to the control system 100 and a distal end D farthest from the control system 100. The distal end D corresponds to the part of the invasive medical device 101 that is first inserted into the patient's anatomical structure in an invasive medical procedure.

[0015]

[0024] The control system 100 includes a medical imaging system 120, a motion detector 130, a workstation 140, a robot 160, and an artificial intelligence controller 180. The workstation 140 includes a controller 150, an interface 153, a monitor 155, and a touch panel 156. The controller 150 includes a memory 151 for storing instructions and a processor 152 for executing the instructions. The interface 153 interfacially connects the monitor 155 to the body of the workstation 140. The artificial intelligence controller 180 includes a memory 181 for storing instructions and a processor 182 for executing the instructions for implementing one or more aspects of the methods described herein.

[0016]

[0025] The characteristics of the invasive medical device 101 affect how the invasive medical device 101 moves with respect to both expected operation and unexpected and / or unintended operations. For example, a flexible guidewire behaves differently from a rigid guidewire with respect to both expected operation and unexpected and / or unintended operations. Thus, the characteristics of the invasive medical device 101 are used as one of the criteria or a plurality of criteria for compensating for unintended operations. Examples of the invasive medical device 101 include a guidewire, a catheter, a microcatheter, and a sheath.

[0017]

[0026] The medical imaging system 120 is an interventional X-ray imaging system. The interventional X-ray imaging system includes an X-ray tube adapted to generate X-rays and an X-ray detector configured to acquire time-series X-ray images such as fluoroscopic images. Examples of such X-ray imaging systems include digital radiography fluoroscopy systems such as Philips' ProxiDiagnost, stationary C-arm X-ray systems such as Philips' Azurion, and mobile C-arm X-ray systems such as Philips' Veradius.

[0018]

[0027] The medical imaging system 120 comprises an image processing controller, which is configured to receive fluoroscopic images acquired during an interventional medical procedure and output a segmentation of the interventional device. The image processing controller may be implemented by / as the controller 150 shown in FIG. 1 or by / as another controller directly integrated into the medical imaging system 120.

[0019]

[0028] The segmentation of the images generated by the medical imaging system 120 generates a display of anatomical features and the surface of structures such as the intervention medical device 101. The segmentation display is composed of, for example, a set of points at three-dimensional (3D) coordinates on the surface of the structure and planar segments of triangles defined by connecting groups of three adjacent points, such that the entire structure is covered by a mesh of non-intersecting planar triangles. The three-dimensional model of the intervention medical device 101 is obtained by segmentation. Also, the segmentation is displayed as a binary mask, the (x, y) coordinates of the intervention medical device 101 in the image space, a two-dimensional spline, or a wireframe model. The segmentation is calculated by thresholding, template matching, active contour modeling, neural network-based segmentation methods, and other segmentation methods. Segmentation is provided for a set of X-ray images generated by an X-ray imaging system or a three-dimensional ultrasound volume generated by an ultrasound imaging system.

[0020]

[0029] The robot 160 is used to control the movement of the intervention medical device 101 under the control of an operator. The operation at the proximal end of the intervention medical device 101 is detected from the operation of the robot 160 when the robot 160 controls the intervention medical device 101.

[0021]

[0030] The artificial intelligence controller 180 includes a plurality of controllers and implements the first artificial intelligence and the second artificial intelligence described herein. The artificial intelligence implemented by the artificial intelligence controller 180 is obtained as a result of training in a dedicated training environment. The artificial intelligence controller 180 is provided separately from the other components of the control system 100 in FIG. 1.

[0022]

[0031] The artificial intelligence controller 180 may be a neural network controller and is used during the application phase of an interventional medical procedure. The artificial intelligence controller 180 is configured to receive motion information at the proximal end of the interventional medical device 101. Also, the artificial intelligence controller 180 is configured to receive from the medical imaging system 120 a fluoroscopic image of the interventional medical device 101 and / or a segmentation display in the fluoroscopic image. Also, the artificial intelligence controller 180 receives the type of the interventional medical device 101 from, for example, a drop-down menu provided via the monitor 155 or from an automatic detection of the interventional medical device 101. The interventional medical device is automatically detected using object detection and classification from an image of the interventional medical device 101 taken by a surgical camera before the interventional medical device 101 is inserted into the patient on the image being captured. The artificial intelligence controller 180 optionally operates based on constraints obtained from the fluoroscopic image and / or the segmentation display of the interventional medical device 101. Constraints that are also used as inputs by the artificial intelligence controller include the length of the interventional medical device 101, the maximum allowable curvature of the interventional medical device 101, and the predicted and / or observed motion transmitted in the longitudinal direction of the interventional medical device 101 visible within the fluoroscopic FOV.

[0023]

[0032] The application results of the first artificial intelligence and the second artificial intelligence by the artificial intelligence controller 180 are predictions of the approximate estimated positions of locations where unexpected behaviors such as buckling occur outside the fluoroscopy FOV, based on the discrepancy between the predicted motion and the observed motion within the fluoroscopy FOV. By identifying unexpected behaviors in the device motion within the fluoroscopy FOV, it aids in identifying possible unintended behaviors that occur outside the fluoroscopy FOV. Another result of the application of the first artificial intelligence and the second artificial intelligence by the artificial intelligence controller 180 is to generate a warning when the agreement between the predicted motion and the observed motion falls outside the normal range. The warning generated using the trained artificial intelligence implemented by the artificial intelligence controller 180 helps prevent the use of excessive force at the proximal end of the intervention medical device 101 when the expected motion is not observed at the distal end of the intervention medical device 101. Similarly, this serves to prevent harmful events such as blood vessel cutting or perforation, pseudoaneurysm, vasoconstriction, and the accidental removal of part of the lesion, as well as other undesirable results such as guidewire breakage.

[0024]

[0033] Although not shown, the control system 100 of FIG. 1 also includes a feedback controller for alerting a physician when unexpected behavior outside the FOV is predicted. The feedback controller generates warnings such as an alarm sound, a printed message on the fluoroscopy display, a tactile feedback to the proximal end of the interventional medical device 101, or a robotic control guidance. The robotic control guidance provides a displayed instruction for modifying the operation at the proximal end of the interventional medical device 101. The displayed instructions include, for example, proposals to move the robotic control forward, backward, or laterally, proposals for the operation of the touch panel or joystick of the Corindus CorPath, proposals for the rotation of the knob for the steerable sheath, the steerable guide catheter, or the transesophageal echocardiogram (TEE) probe. Also, the feedback controller provides robotic control guidance in a closed-loop system, for example, by sending a command to automatically retreat the autonomous robot. Further, the feedback controller provides robotic control guidance to prevent the cooperative control robot from advancing further when the forward movement is causing buckling.

[0025]

[0034] The control system 100 has been mainly described in the context of an X-ray imaging system, but the control system 100 includes or is incorporated into both an interventional ultrasound imaging system and both stationary and mobile interventional X-ray imaging systems. The control system 100 is used for various fluoroscopy-based interventional medical procedures, including but not limited to interventional vascular procedures.

[0026]

[0035] FIG. 2 shows a method of predictive motion mapping for a flexible device according to an exemplary embodiment.

[0027]

[0036] In S210, the method of FIG. 2 starts by training an artificial intelligence. The trained artificial intelligence includes a first artificial intelligence and a second artificial intelligence that are trained using different inputs to generate different outputs. Further, the output from the first artificial intelligence is the input to the second artificial intelligence. Further, the first prediction or anticipation by the first artificial intelligence is output from the first artificial intelligence, input into the second artificial intelligence, and the second artificial intelligence outputs a second prediction or anticipation based on using the first prediction or anticipation from the first artificial intelligence as an input. The ground truth information of the approximate estimation position of the unexpected behavior of the intervention-type medical device is used for the training of the artificial intelligence, and the trained artificial intelligence is used after the artificial intelligence is deployed. The features of the first artificial intelligence and the second artificial intelligence will be described in detail in the following paragraphs. The training of the artificial intelligence in S210 is all performed before the artificial intelligence is deployed. In one embodiment, the adaptive artificial intelligence uses post-deployment feedback for self-improvement via reinforcement learning or other learning methods.

[0028]

[0037] The artificial intelligence controller 180 in FIG. 1 performs training on the first trained artificial intelligence and the second trained artificial intelligence. For example, in a plurality of training sessions for a plurality of intervention-type medical devices, the artificial intelligence controller 180 inputs at least one position at the distal end of the intervention-type medical device, detects the operation at the proximal end of the intervention medical device, and detects the operation along the intervention-type medical device toward the distal end of the intervention-type medical device caused by the operation at the proximal end of the intervention-type medical device. In a plurality of training sessions for a plurality of intervention-type medical devices, the artificial intelligence controller 180 is further configured to input the type of the intervention-type medical device, the anatomical structure or the type of treatment, or other context information that varies depending on different intervention-type medical procedures. The position of the distal end of the intervention-type medical device in the training is obtained from an image such as that derived from a medical image by a medical imaging system. The plurality of training sessions also include predicting a predicted operation along the intervention-type medical device toward the distal end of the intervention-type medical device based on at least one position of the distal end of the intervention-type medical device and the detected operation at the proximal end of the intervention-type medical device. The plurality of training sessions also include detecting the actual operation along the intervention-type medical device toward the distal end of the intervention-type medical device and determining a loss based on the difference between the predicted operation and the actual operation. The first trained artificial intelligence establishes the relationship between the operation at the proximal end of the intervention-type medical device and the operation along the intervention-type medical device toward the distal end of the intervention-type medical device, and the first artificial intelligence is updated based on each loss determined based on the difference between the predicted operation and the detected actual operation.

[0029]

[0038] After the training in S210, the artificial intelligence is provided for use. Then, the first artificial intelligence is implemented by the artificial intelligence controller 180 in FIG. 1.

[0030]

[0039] For an intervention-type medical treatment, a first artificial intelligence is further trained for different types of intervention-type medical devices 101. The first trained artificial intelligence optionally inputs and operates based on at least one of the type of the intervention-type medical device 101, the type of the intervention-type medical treatment, anatomical landmarks, or at least one physical characteristic of the patient. A clinician is provided with a drop-down menu for selecting the type of the intervention-type medical device 101. The predicted motion transmitted through the intervention-type medical device 101 near the distal end of the intervention-type medical device 101 is predicted further based on the selected type of the intervention-type medical device. Similarly, the prediction of the predicted motion may be further based on the anatomical structure of the patient in the intervention-type medical treatment, the arrangement of the medical imaging system, or the physical characteristics of the intervention-type medical device 101. Alternatively, before the intervention-type medical treatment, for example, before the intervention-type medical device 101 is inserted into the patient, the type of the intervention-type medical device 101 targeted by the first artificial intelligence is automatically selected by detecting and classifying objects. The detection and classification are performed based on images captured by a camera in the operating room. Alternatively, the detection and classification are performed using a model trained by machine learning to detect and classify different types of intervention-type medical devices. The training dataset of machine learning used for creating the model includes training instances including X-ray images of a plurality of different intervention-type medical devices. The training data includes only the normal or expected operations at the distal ends of a plurality of different intervention-type medical devices, whereby the artificial intelligence learns to predict the normal operations at the distal ends of different intervention-type medical devices, and during inference, if the subsequent observed motion is not similar to the predicted normal operation, an alarm or alert is generated and issued. The training data is collected using shape detection techniques such as FORS.Since FORS provides 3D shape information along the length of the device, it enables confirmation that the data includes expected operations and does not include unexpected operations such as buckling.

[0031]

[0040] In an operation during an interventional medical procedure, a first artificial intelligence is also implemented based on additional context information such as a target region or anatomical structure, segmentation of surrounding anatomical structures, and the pose of the C-arm with the target region, to enable the first artificial intelligence to learn when foreshortening should be expected. Further, the first artificial intelligence is subject to constraints on the output, such as the length of the interventional medical device 101 in a fluoroscopic image, or the maximum allowable curvature of the interventional medical device 101.

[0032]

[0041] The first artificial intelligence may be a neural network such as a convolutional neural network, an encoder-decoder network, a generative adversarial network, a capsule network, a regression network, a reinforcement learning agent, etc., and uses the motion information at the proximal end of the interventional medical device 101 and the fluoroscopic image at the initial time t to predict the motion or motion field transmitted in the length direction of the interventional medical device 101 visible within the fluoroscopic field of view (FOV). The observed motion or motion field between the fluoroscopic images at times t and t + n is compared with the motion predicted by the first artificial intelligence to learn the expected range of motions observable in the fluoroscopic FOV. Time t + n may be after a specific motion at the proximal end is completed, or may be any other arbitrary time, such as when a specific motion at the proximal end is occurring. Mean squared error, mean absolute error, or Huber loss, or two motion vectors (R 2 、R 3)Any loss that calculates the difference between, for example, geodesic loss or other loss (loss) functions is calculated to compare the predicted motion and the observed (ground truth) motion. The operation is represented in vector parameter display, non-vector parameter display, and / or an operation field. The parameter display can be in the form of Euler angles, quaternions, matrices, exponential maps, and / or angle axes representing rotation and / or translation (e.g., including the direction and magnitude of translation and rotation).

[0033]

[0042] In S220, the method of FIG. 2 includes identifying the operation at the proximal end of the intervention type medical device. The operation at the proximal end can include forward movement, lateral movement, and rotation along the axis. The operation at the proximal end is an operation caused by the user or the robot, and includes any operation caused by the user or the robot to control the intervention type medical device 101. The operation information at the proximal end of the intervention type device is obtained from the detection device. Examples of detection devices that can capture and provide such operation information include device trackers, inertial measurement unit (IMU) sensors, monocular or stereo camera systems acting as optical tracking systems, linear encoders, torque encoders, or optical encoders. Examples of device trackers include optical tracking detection systems, optical tracking systems, electromagnetic tracking systems, or optical shape detection mechanisms. Examples of IMU sensors include sensors that measure angular velocity, force, and optionally magnetic field by components such as accelerometers, gyroscopes, and optionally magnetometers. Examples of linear encoders include optical linear encoders, magnetic linear encoders, and capacitive inductive linear encoders.

[0034]

[0043] In S225, the method of FIG. 2 includes obtaining a set of medical images of the interventional medical device. The set of medical images obtained in S225 is obtained by the medical imaging system 120. The set of medical images obtained in S225 is used to obtain at least one position of the distal end of the interventional medical device 101 from the image of the distal end of the interventional medical device 101. The set of medical images is a plurality of fluoroscopic images of a part of the interventional medical device within the field of view of the medical imaging system. The set of medical images obtained in S225 is a part of the interventional medical device near the distal end. The embodiment based on FIG. 2 includes that the segmentation display of the interventional medical device is used as an input to the first artificial intelligence applied in S230, for example, when used as an input to the first artificial intelligence applied in S230, obtaining an image of the distal end of the interventional medical device. In FIG. 4B described later, the image of the distal end is represented as the fluoroscopic frame f t as shown. The set of medical images includes a single or time-series fluoroscopic image including the distal end of the interventional medical device 101. The set of medical images is automatically segmented by an image processing controller using methods such as threshold processing, template matching, active contour modeling, multi-scale ridge enhancement filter, or a segmentation algorithm based on deep learning.

[0035]

[0044] In S230, the method of FIG. 2 includes applying a trained first artificial intelligence to a specified operation at the proximal end of the interventional medical device and a medical image of the distal end of the interventional medical device at the time when the operation is applied to the proximal end of the interventional medical device. The trained first artificial intelligence is an artificial intelligence trained to discover the correlation between the operation applied at the proximal end of the interventional medical device and the operation received at the distal end of the interventional medical device observed in the set of interventional medical images.

[0036]

[0045] In S240, the first artificial intelligence predicts an operation along the intervention-type medical device towards the distal end based on the operation specified at the proximal end of the intervention-type medical device 101 in S220 and the image of the intervention-type medical device near the distal end in S225. The first artificial intelligence is implemented by receiving, in S225, a fluoroscopic image of a segmentation display of the intervention-type medical device that covers an image of the intervention-type medical device 101 without unexpected / unintentional behaviors such as buckling in the initial state. Since the first artificial intelligence is trained before the intervention-type medical treatment, the first artificial intelligence uses the initial information of the intervention-type medical device from the segmentation display as a criterion for determining the normal or expected proximal-distal motion mapping.

[0037]

[0046] In S250, the method of FIG. 2 includes acquiring an image of the intervention-type medical device from the medical imaging system. The image of the intervention-type medical device acquired in S250 is an image of the distal end of the intervention-type medical device and / or an image near the distal end of the intervention-type medical device.

[0038]

[0047] In S255, the method of FIG. 2 includes segmenting the intervention-type medical device within the image by the medical imaging system. As a result of the segmentation, a segmentation display of the intervention-type medical device is obtained. S255 is optional and is also executed on the image of the intervention-type medical device near the distal end, which is acquired in S225 and input to the first artificial intelligence trained in S230.

[0039]

[0048] In S257, the actual operation is detected from the image from the medical imaging system.

[0040]

[0049] In S260, the detected actual operation of the intervention medical device is compared with the predicted operation of the intervention medical device. The predicted operation of the intervention medical device compared in S260 in FIG. 2 is the operation predicted in S240.

[0041]

[0050] In S270, it is determined whether the actual operation deviates from the predicted operation. The deviation may be identified by a binary classification process, or may be based on one or more thresholds, a scoring algorithm, or other processes for determining whether the actual operation of the intervention medical device is within the range predicted by the predicted operation.

[0042]

[0051] If the actual operation does not deviate from the predicted operation (S270 = No), an alarm is not generated. If the actual operation deviates from the predicted operation (S270 = Yes), an alarm is generated in S280.

[0043]

[0052] Further, in S271, the method of FIG. 2 includes predicting a rough estimated position of the operation transmitted by the intervention medical device outside the field of view of the image from the intervention medical device. Further, the second artificial intelligence predicts the prediction reliability of the predicted rough estimated position. The rough estimated position predicted in S271 is predicted by the second artificial intelligence described herein. The second artificial intelligence is implemented by the artificial intelligence controller 180 and implements a position estimation neural network. The second artificial intelligence is configured to receive the predicted device operation and the observed device operation from the fluoroscopic image. The data for training the second artificial intelligence in S210 to predict the rough estimated position of the operation occurring outside the FOV of the medical imaging system is obtained using a shape detection technique such as FORS.

[0044]

[0053] The second artificial intelligence is implemented by a trained neural network such as a convolutional neural network, an encoder-decoder network, an adversarial generative network, a capsule network, a regression network, a reinforcement learning agent, etc. The second artificial intelligence uses the predicted motion and the observed motion at the distal end as inputs to predict whether unexpected and / or unintended behaviors such as buckling are occurring outside the fluoroscopy FOV and where they are occurring, and compares that prediction with the ground truth information of the ground truth estimated position obtained in the training in S210 such as from FORS. By calculating a loss function such as mean squared error, mean absolute error, or Huber loss, the predicted estimated position is compared with the ground truth estimated position based on the ground truth information. The second artificial intelligence generates a warning if an unexpected / unintended behavior is predicted. The warning is generated based on the presence or absence of the unintended behavior predicted in the manner described herein.

[0045]

[0054] In S281, a display is generated for the predicted approximate estimated position transmitted by the intervention medical device outside the FOV of the image.

[0046]

[0055] Figure 3 shows another method for predicting motion mapping of a flexible device according to another exemplary embodiment.

[0047]

[0056] In S310, the method of Figure 3 includes inputting the motion detected at the proximal end of the intervention medical device.

[0048]

[0057] In S320, at least one position of the distal end of the intervention medical device is detected.

[0049]

[0058] In S330, the first artificial intelligence is trained to predict the motion along the intervention medical device towards the distal end.

[0050]

[0059] In S360, the operation along the intervention medical device towards the distal end is predicted based on the operation at the proximal end and the medical image of the distal end of the intervention medical device before applying the operation at the proximal end of the intervention medical device.

[0051]

[0060] In S370, the actual operation along the intervention medical device towards the distal end is detected. The actual operation is detected from a partial medical image or a segmentation display of the intervention medical device within the field of view of the medical imaging system.

[0052]

[0061] In S380, the method of FIG. 3 includes determining a loss function based on the difference between the predicted operation and the actual operation towards the distal end of the intervention medical device.

[0053]

[0062] In S385, the first neural network is updated based on the determined loss function, and the process returns to S330.

[0054]

[0063] In the embodiment of FIG. 3, the first neural network is updated in S385 when the first neural network is trained. In one embodiment, if the data generated during operation represents a normal or expected proximal-distal operation mapping and can be used with high confidence as ground truth, the first neural network is updated in S385 using this data after the first neural network has become operational.

[0055]

[0064] FIG. 4A shows a hybrid process for the predicted operation mapping of a flexible device according to an exemplary embodiment.

[0056]

[0065] In FIG. 4A, the first artificial intelligence 410A and the second artificial intelligence 415A are trained using a first loss function (loss function #1) and a second loss function (loss function #2) based on inputs including operations at the proximal and distal ends of an intervention-type medical device. The inputs to the first artificial intelligence 410A and the second artificial intelligence 415A during training are described for purposes of illustration with respect to the corresponding features of FIG. 4B, as explained below.

[0057]

[0066] FIG. 4B shows another hybrid process for predictive motion mapping of a flexible device according to an exemplary embodiment.

[0058]

[0067] Figure 4B schematically depicts a process and system for proximal-to-distal prediction motion mapping. An interventional medical device 401, such as a guide wire, and a medical imaging system 420, such as a fluoroscopic X-ray medical imaging system, are used for training artificial intelligence, such as in a control environment. The interventional medical device 401 includes a first region visible in medical imaging and a second region not visible in medical imaging. The first region and the second region change during operation as the FOV of the medical imaging system 420 changes. During operation, when the field of view of the medical imaging changes, the first region and the second region change. A first artificial intelligence is trained by using the operation at the proximal end of the interventional medical device 401, the medical image at the distal end of the interventional medical device 101 when the operation is applied to the proximal end of the interventional medical device 101, and predicting the resulting operation at the distal end and comparing the operation with the observed operation at the distal end, so as to establish the relationship between the operation at the proximal end of the interventional medical device 401 and the resulting operation at the distal end of the interventional medical device 401. A second artificial intelligence is trained by using the observed operation and the predicted operation at the distal end to predict the approximate estimated position of the unintended behavior of the interventional medical device 401 and comparing the predicted approximate estimated position of the unintended behavior with the ground truth approximate estimated position of the unintended behavior. The training of the second neural network 415B uses the ground truth information of the approximate estimated position at the distal end of the interventional medical device, whereby the learning by training is used after the second neural network 415B is deployed.

[0059]

[0068] In Figure 4B, the fluoroscopic frame f tThe segmentation 420 of the interventional medical device 401 in [description] is provided as an input to the first neural network 410B along with the operations applied at the proximal end of the interventional medical device 401. For the sake of clarity, the segmentation of the interventional medical device 401 in FIG. 4B is performed for a first region that includes a part of the interventional medical device 401 within the field of view (FOV) of the medical imaging system. Another part of the interventional medical device 401 is not within the field of view (FOV) of the medical imaging system. The first neural network 410B outputs a point-by-point motion estimation that propagates in the longitudinal direction of the segmentation display of the interventional medical device 401. The point-by-point motion estimation is compared with the observations calculated from the segmentation display of the interventional medical device 401 in the subsequent fluoroscopic frame f t+n The segmentation of the interventional medical device 401 in [description] is compared with the observations calculated from the segmentation display of the interventional medical device 401 in [description]. In other words, the first neural network 410B in FIG. 4B learns the correlation of how the operation at the proximal end of the interventional medical device 401 results in the operation at the points along the distal end of the interventional medical device 401.

[0060]

[0069] Also, in FIG. 4B, the estimated motion for each predicted point transmitted in the longitudinal direction of the segmentation display of the intervention type medical device, and the actual motion observed in the segmentation display are input to the second neural network 415B. Just to be sure, the input to the second neural network 415B is about the estimated motion and the actual motion of the intervention type medical device 401 in a first region including a part of the intervention type medical device 401 in the field of view (FOV) of the medical imaging system. The output of the second neural network 415B is a prediction of the approximate estimated position of the unintended motion in a second region including a part of the intervention type medical device 401 outside the field of view of the medical imaging system. The approximate estimated position estimated in the second region is compared with the ground truth estimated position in the second region obtained from a mechanism such as a shape detection technique. For example, the ground truth estimated position is obtained via a shape detection technique such as Fiber Optic RealShape (FORS) by Philips.

[0061]

[0070] As described above, in FIG. 4B, the first neural network 410B is trained to output an estimated motion for each point along the longitudinal direction of the intervention medical device 401 toward the distal end based on the input of the motion at the proximal end of the intervention medical device 401 and the segmentation display of the intervention medical device 401. The training of the first neural network 410B is based on feeding back a first loss function (loss function 1) that reflects the difference between the predicted motion and the observed motion at the distal end of the intervention medical device 401. The second neural network 415B is trained to output a rough estimated position of the unintended behavior of the intervention medical device 401 based on the motion estimation for each point and the motion that can actually be observed within the field of view of the medical imaging system. The training of the second neural network 415B is based on feeding back a second loss function (loss function 2) that reflects the difference between the output of the rough estimated position and the ground truth estimated position of the unintended behavior. During the training of FIG. 4B, the predicted rough estimated position of the unintended behavior is determined through the second loss function using, for example, the actual estimated position by optical shape sensing. After the second neural network is trained to an acceptable accuracy, the use of the predicted estimated position during training is applied during operation even when the actual estimated position is not used. As a result, the unintended behavior of the intervention medical device 401 can be predicted and roughly located.

[0062]

[0071] FIG. 5 shows a method for predicting the motion mapping of a flexible device according to a representative embodiment.

[0063]

[0072] In FIG. 5, the training of the first artificial intelligence and the second artificial intelligence is described. In S510, the observed motion at the proximal end of the intervention medical device is input as a first input to the first neural network.

[0064]

[0073] In S520, the segmentation of the intervention type medical device at the distal end is input as a second input to the first neural network.

[0065]

[0074] In S525, the first neural network is applied to the inputs from S510 and S520.

[0066]

[0075] In S530, the predicted operation at the distal end of the intervention type medical device is output from the first neural network.

[0067]

[0076] In S540, the observed operation at the distal end of the intervention type medical device is compared with the predicted operation at the distal end of the intervention type medical device to generate a first loss function.

[0068]

[0077] In S545, the first neural network is updated. The process returns to S525 and continues the process of training the first neural network until the process ends.

[0069]

[0078] In S550, the predicted operation at the distal end of the intervention type medical device from the output of the first neural network is input as a first input to the second neural network.

[0070]

[0079] In S560, the observed operation at the distal end of the intervention type medical device is input as a second input to the second neural network.

[0071]

[0080] In S565, the second neural network is applied.

[0072]

[0081] In S570, the approximate estimated position of the unintended behavior outside the field of view of the imaging device is output by the second neural network.

[0073]

[0082] In S580, the ground truth estimation position of the unintended behavior of the intervention medical device is compared with the predicted approximate estimation position of the unintended behavior to generate a second loss function.

[0074]

[0083] In S585, the second loss function is fed back to update the second neural network. After S585, the process of training the second neural network returns to S565.

[0075]

[0084] FIG. 6A is a diagram showing a hybrid process for predictive motion mapping for a flexible device according to an exemplary embodiment.

[0076]

[0085] In FIG. 6A, a first artificial intelligence 610A and a second artificial intelligence 615A are used in the operations that generate the outputs described herein. The inputs to the first artificial intelligence 610A and the second artificial intelligence 615A during operation are illustrated as examples for the corresponding features of FIG. 6B as described below.

[0077]

[0086] FIG. 6B shows another hybrid process for predictive motion mapping for a flexible device according to an exemplary embodiment.

[0078]

[0087] In FIG. 6B, another proximal - distal predictive motion mapping process and system are schematically represented. The hybrid process and system of FIG. 6B are used during an intervention medical procedure. Similar to the hybrid process of FIG. 6B, any fluoroscopy frame f tRegarding the segmentation of the intervention medical device 601 in the first region within the field of view of medical imaging, it is provided as an input to the first neural network 610B along with the operation applied at the proximal end of the intervention medical device 601. The first neural network 610B outputs an operation estimation for each point along the length direction of the segmentation display of the intervention medical device 601. The estimation by the first neural network 610B is compared with the observation calculated from the segmentation display of the intervention medical device 601 in the first region within the field of view of the medical imaging system in the subsequent fluoroscopy frame f t+n It is compared with the observation calculated from the segmentation display of the intervention medical device 601 in the first region within the field of view of the medical imaging system in the fluoroscopy frame f t+n . When the two operations, i.e., the estimated operation and the observed operation, do not match, they serve as the input to the second neural network 616B that predicts the approximate estimated position of where there may be an unintended operation in the second region outside the fluoroscopy FOV.

[0079]

[0088] FIG. 6C shows a method for predictive motion mapping of a flexible device according to an exemplary embodiment.

[0080]

[0089] In FIG. 6C, the observed operation at the proximal end of the intervention medical device is input as the first input to the first neural network at S610.

[0081]

[0090] At S620, the segmentation of the intervention medical device at the distal end of the intervention medical device is input as the second input to the first neural network. The segmentation display is provided as a binary mask of the segmentation display of the intervention medical device in the fluoroscopy image.

[0082]

[0091] At S625, the first neural network is applied to the first input at S610 and the second input at S620.

[0083]

[0092] In S630, a predicted motion at the distal end of an interventional medical device is output from a first neural network as a predicted motion. The first neural network may be a trained encoder-decoder network. The predicted motion is a motion along the longitudinal direction of the interventional medical device towards the distal end and is output by the trained encoder-decoder network.

[0084]

[0093] In S650, the predicted motion at the distal end of the interventional medical device output from the first neural network is input as a first input to a second neural network.

[0085]

[0094] In S660, an observed motion at the distal end of the interventional medical device is input as a second input to the second neural network. The second neural network may be a trained convolutional neural network.

[0086]

[0095] In S665, the second neural network is applied to the predicted motion in S650 and the observed motion in S660.

[0087]

[0096] In S670, the second neural network outputs a rough estimation position of an unintended behavior outside the FOV of the medical imaging system. The second neural network locates an unexpected / unintended behavior in the interventional medical device 101 outside the fluoroscopy FOV based on the discrepancy between the predicted motion and the observed motion at the distal end of the interventional medical device within the FOV of the medical imaging system. The prediction is used to generate the above alarm.

[0088]

[0097] As described in the above embodiments, one or more deep learning algorithms are trained to learn the relationship or mapping between the actions applied at the proximal end of the intervention medical device and the actions observed at the distal end of the intervention medical device. The captured input actions include manual actions or mechanical actions such as robotic actions or robot-assisted actions, and can be rotations and / or translations. In an alternative embodiment, the deep learning algorithm also learns the proximal-distal mapping of the velocity field, acceleration, inertia, spatial configuration, tangential (angular) actions, and linear velocity or linear acceleration at multiple points. The learning by the deep learning algorithm takes into account specific parameters of the intervention medical device. During the procedure, when the actions applied to the intervention medical device at the proximal end and the medical image of the distal end of the intervention medical device at the time when the actions are applied to the proximal end of the intervention medical device are provided, the control system 100 estimates the actions of the intervention medical device at the distal end. The control system 100 also learns to associate the difference between the predicted device actions and the observed device actions at the distal end with different unobserved device behaviors that occur outside the fluoroscopy FOV. The control system 100 then alerts the physician about possible unintended behaviors in the intervention medical device outside the FOV of the medical imaging system, thereby preventing possible vascular damage or other undesirable results.

[0089]

[0098] In one embodiment, consistent with the above teachings, the deep learning model is trained to predict the actions at the distal end of the intervention medical device 101 from the two-dimensional coordinates of the segmentation display of the intervention medical device 101, or from the two-dimensional coordinates of the spline that conforms to the segmentation display of the intervention medical device in the fluoroscopy image.

[0090]

[0099] In another embodiment, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) network, a temporal convolutional network (TCN), or a transformer is used to observe the segmentation display of the intervention medical device 101 in a plurality of fluoroscopy frames (t_0 to t_n) in order to better inform the motion prediction in frame t_n.

[0091]

[0100] In another embodiment, the deep learning model is trained to predict the position of the segmentation display of the intervention medical device in frame t_(n + 1), which is obtained from the segmentation of the intervention medical device either only in frame t_n or in any of frames t_0 to t_n. The prediction is directly compared with the device observed in fluoroscopy frame t_(n + 1).

[0092]

[0101] In another embodiment, a 3D model, or a set of parameters, rules, or characteristics of a known intervention medical device 101 is used to inform the prediction of the motion or speed of the intervention medical device 101.

[0093]

[0102] In another embodiment, a deep learning algorithm is trained to learn the proximal-distal mapping of device-specific parameters. Examples of device-specific parameters include velocity fields, accelerations, inertia, spatial configurations, and tangents / angles over a plurality of points, and linear velocity or acceleration. In this embodiment, the predicted parameters are compared against the measured parameters.

[0094]

[0103] In another embodiment, a machine learning algorithm uses the predicted motion information and the observed motion information to classify the observation result as normal or abnormal. Examples of machine learning algorithms include one-class support vector machine (SVM) classification or deep learning-based classification. In this embodiment, a warning is generated when an abnormality is detected.

[0095]

[0104] In another embodiment, the deep learning network is trained to predict the operation at the distal end of the interventional medical device from the ultrasound image at the distal end of the interventional medical device. The input to the deep learning network is provided from an ultrasound image, a binary mask of the segmentation display of the interventional medical device in the ultrasound, the two-dimensional (x, y) coordinates of the segmentation display of the interventional medical device 101 in the ultrasound, or the two-dimensional (x, y) coordinates of the spline that fits the segmentation display of the interventional medical device 101 in the ultrasound.

[0096]

[0105] In yet another embodiment, the deep learning network is further trained to learn the confidence of the predicted operation at the distal end of the interventional medical device based on agreement with the ground truth during training, or any other method of determining confidence or uncertainty. The control system 100 learns the type of input operation at the proximal end of the interventional medical device 101 that generates a reliable estimate at the distal end, or learns the type of fluoroscopic image associated with a reliable operation prediction. For example, the control system 100 learns that an image showing foreshortening does not generate a very reliable operation estimate at the distal end. The confidence of the predicted operation at the distal end predicted by the first deep learning network is additionally input to the second network to predict the approximate estimated position of the unintended behavior transmitted by the interventional medical device 101 outside the field of view of the medical imaging system. Similarly, the second deep learning network is trained to further predict the confidence of the predicted approximate estimated position of the unintended behavior transmitted by the interventional medical device 101 outside the field of view of the medical imaging system.

[0097]

[0106] In yet another embodiment, the intravascular robotic system measures the force applied at the distal end of the catheter and displays the measured force value on the console or incorporates the measured force value into a control loop. This function continuously alerts the clinician of the risk of ongoing force application, thereby reducing the likelihood of perforation or other damage to the blood vessel wall. In this embodiment, detection of abnormal or unintended device behavior, such as buckling, is incorporated into either the safety mechanism or the control loop of the robotic system. For example, if buckling is predicted at the end effector (distal portion), the risk of blood vessel wall perforation is high and the robotic actuator will decelerate or an emergency stop will be triggered. The user is notified on the console and requested to take corrective action. Alternatively, in the case of a semi-autonomous or fully autonomous robot, the control system 100 automatically retracts the end effector, maneuvers in a different direction, and makes a re-approach for cannula insertion. If buckling occurs in the medical section of the guide wire, the control system 100 adjusts the settings of the controller 150, such as the PID controller parameters including gain and motor speed, in the background using either pre-programmed rules or a complex prediction model. The user is notified only if the adjustment of the controller fails or has no effect, thus avoiding operator overload, cognitive burden, and a decrease in trust in the robotic system. The robotic system also learns the operator's preferences and notifies the operator only if buckling occurs in a specific area, at a specific intensity or frequency, or for a specific period of time.

[0098]

[0107] In yet another embodiment, complex inputs at the proximal end of the intervention medical device 101 in the form of touch panel or joystick operations, such as those that control Corindus CorPath, or knob rotations, such as those that control an actuatable sheath, an actuatable guide catheter, a TEE probe, etc., are incorporated. When the control system 100 detects an unintended behavior in the intervention medical device, the control system 100 proposes a mechanism for eliminating the unintended behavior associated with the input device. Examples of input devices include touch panels, joysticks, and knobs.

[0099]

[0108] FIG. 7 shows a computer system in which a method for predictive motion mapping of a flexible device is implemented according to another representative embodiment.

[0100]

[0109] The computer system 700 of FIG. 7 shows a complete set of components of a communication device or computer device. However, the "controller" described herein is implemented with a set of components fewer than the set of components of FIG. 7, such as a combination of a memory and a processor. The computer system 700 includes some or all of the elements of one or more of the constituent devices in the system for predictive motion mapping of the flexible device herein, but such devices do not necessarily include one or more of the elements described for the computer system 700 and may include other elements not described.

[0101]

[0110] Referring to FIG. 7, computer system 700 includes a set of executable software instructions that cause computer system 700 to execute any of the methods or computer-based functions disclosed herein. Computer system 700 operates as a stand-alone device or is connected to other computer systems or peripheral devices, for example, using network 701. In an embodiment, computer system 700 performs logical processing based on digital signals received via an analog-to-digital converter.

[0102]

[0111] In a network deployment, computer system 700 operates within the capabilities of a server, or as a client user computer in a server-client user network environment, or as a peer computer system in a peer-to-peer (or distributed) network environment. Computer system 700 can be implemented as or incorporated into various devices such as the controller 150 of FIG. 1, a desktop computer, a mobile computer, a personal computer (PC), a laptop computer, a tablet computer, or any other machine capable of executing a set of (sequence or other) software instructions that specify the actions to be taken by that machine. Computer system 700 can likewise be implemented as or incorporated into a device within an integrated system that includes additional devices. In one embodiment, computer system 700 can be implemented using an electronic device that provides voice, video, or data communication. Further, although computer system 700 is shown in singular, the term "system" should also be regarded as including any plurality of systems or subsystems that execute, individually or in combination, a set or sets of software instructions that perform one or more computer functions.

[0103]

[0112] As shown in FIG. 7, computer system 700 includes a processor 710. The processor 710 is considered a representative example of the processor 152 of the controller 150 in FIG. 1 and executes instructions to implement some or all aspects of the methods and processes described herein. The processor 710 is tangible and non-transitory. As used herein, the term "non-transitory" should be construed as a characteristic of a state that persists over a period of time, rather than as an invariant characteristic of a state. The term "non-transitory" specifically negates transient characteristics such as the characteristics of a carrier wave or a carrier signal, or other forms that exist only temporarily at any location and at any time. The processor 710 is a product and / or a mechanical component. The processor 710 is configured to execute software instructions that perform the functions described in various embodiments herein. The processor 710 may be a general-purpose processor or may be part of an application-specific integrated circuit (ASIC). The processor 710 may be a microprocessor, a microcomputer, a processor chip, a controller, a microcontroller, a digital signal processor (DSP), a state machine, or a programmable logic device. The processor 710 may be a programmable gate array (PGA) such as a field-programmable gate array (FPGA), or a logic circuit including other types of circuits including discrete gates and / or transistor logic. The processor 710 may be a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or any combination thereof. Further, any processor described herein may include multiple processors, parallel processors, or both. The multiple processors are included in or connected to a single device or multiple devices.

[0104]

[0113] As used herein, the term "processor" encompasses an electronic component capable of executing programs or machine-executable instructions. References to a computing device having a "processor" should be construed to include multiple processors or processing cores, as well as multi-core processors. A processor also refers to a collection of processors, either within a single computer system or distributed across multiple computer systems. Further, the term "computing device" should be construed to include a collection or network of computing devices, each of which includes one or more processors. A program has software instructions that are executed by one or more processors, either within the same computing device or distributed across multiple computing devices.

[0105]

[0114] Computer system 700 further includes main memory 720 and static memory 730, and the memories of computer system 700 communicate with each other and also communicate with processor 710 via bus 708. Either or both of main memory 720 and static memory 730 can be considered representative examples of the memory 151 of controller 150 in FIG. 1 and store instructions used to implement some or all aspects of the methods and processes described herein. The memories described herein are tangible storage media that store data and executable software instructions and are non-transitory while the software instructions are stored. As used herein, the term "non-transitory" should be construed as a property of a state that persists over a period of time rather than as an immutable property of a state. The term "non-transitory" specifically negates transient properties such as the properties of a carrier wave or carrier signal or other forms that exist only temporarily at any place and any time. Main memory 720 and static memory 730 are products and / or machine components. Main memory 720 and static memory 730 are computer-readable media from which data and executable software instructions can be read by a computer (e.g., processor 710). Each of main memory 720 and static memory 730 is implemented as one or more of random access memory (RAM), read-only memory (ROM), flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM®), registers, hard disk, removable disk, tape, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), floppy disk, Blu-ray disc, or any other form of storage medium known in the art. The memory may be volatile or non-volatile, secure and / or encrypted, non-secure and / or unencrypted.

[0106]

[0115] "Memory" is an example of a computer-readable storage medium. Computer memory is any memory directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be interpreted as potentially referring to multiple memories. Memory can be, for example, multiple memories within the same computer system. Memory can also be multiple memories distributed across multiple computer systems or computing devices.

[0107]

[0116] As shown, computer system 700 further includes, for example, a video display unit 750 such as a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid state display, or a cathode ray tube (CRT). Additionally, computer system 700 includes an input device 760 such as a keyboard / virtual keyboard, a touch-sensitive input screen, or voice input using voice recognition, and a cursor control device 770 such as a mouse, a touch-sensitive input screen, or a pad. Also, computer system 700 optionally includes a disk drive unit 780, a signal generation device 790 such as a speaker or a remote control, and / or a network interface device 740.

[0108]

[0117] In one embodiment, as depicted in FIG. 7, the disk drive unit 780 includes a computer-readable medium 782 in which one or more sets of software instructions 784 (software) are embedded. The set of software instructions 784 is read from the computer-readable medium 782 and executed by the processor 710. Further, when executed by the processor 710, the software instructions 784 perform one or more steps of the methods and processes described herein. In one embodiment, all or part of the software instructions 784 reside in the main memory 720, the static memory 730, and / or the processor 710 while being executed by the computer system 700. Further, the computer-readable medium 782 includes the software instructions 784 or receives and executes the software instructions 784 in response to a propagated signal, such that as a result, a device connected to the network 701 communicates voice, video, or data over the network 701. The software instructions 784 are transmitted or received on the network 701 via the network interface device 740.

[0109]

[0118] In one embodiment, dedicated hardware implementations such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays, and other hardware components are constructed to implement one or more of the methods described herein. One or more embodiments described herein function using two or more specifically interconnected hardware modules or devices having related control and data signals that are communicable between and through the modules. Accordingly, the present disclosure encompasses software, firmware, and hardware implementations. None of the claims of this application should be construed as being implemented or implementable solely in software without the use of hardware such as tangible and non-transitory processors and / or memories.

[0110]

[0119] According to various embodiments of the present disclosure, the methods described herein are implemented using a hardware computer system that executes a software program. Further, in exemplary and non-limiting embodiments, the implementation can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing implements one or more of the methods or functions described herein, and the processors described herein are used to support a virtual processing environment.

[0111]

[0120] Thus, the predicted motion mapping of a flexible device provides guidance regarding the actual positioning of an intervention medical device when, for example, the intervention medical device is used in an intervention medical procedure under the guidance of live two-dimensional fluoroscopic imaging. The characteristics of the intervention medical device that can lead to unintended operations can be used as a criterion for compensating for the unintended operations. Similarly, the patient's anatomical structure that can lead to unintended operations can be used as a criterion for compensating for the unintended operations.

[0112]

[0121] The present invention describes a system that learns the range of possible motions or behaviors at the distal end when given a specific motion or action at the proximal end and the current placement or configuration of the distal end of an intervention medical device. Thereby, it can predict the expected motion transmitted in the longitudinal direction of the guide wire visible in fluoroscopy and issue an alert when the observed motion is outside the range of the expected motion. Further, the system can observe the type of unintended motion in the fluoroscopic field of view (FOV) and predict where an unintended or un-intended behavior outside the fluoroscopic FOV is occurring, e.g., whether it is close to or away from the FOV.

[0113]

[0122] Nevertheless, the predictive motion mapping for the flexible device is not limited to application to the specific details described herein. Instead, one or more inputs to the first artificial intelligence and the second artificial intelligence are applicable to additional embodiments that differ from the specific details described for the embodiments herein.

[0114]

[0123] Although the predictive motion mapping for the flexible device has been described with reference to some exemplary embodiments, it is understood that the terms used are for purposes of explanation and illustration, not limitation. Without departing from the scope and spirit of the predictive motion mapping for the flexible device in that regard, changes can be made within the scope of the appended claims, as referred to and modified herein. The predictive motion mapping for the flexible device has been described with reference to specific means, materials, and embodiments, but the predictive motion mapping for the flexible device is not intended to be limited to the specific ones disclosed. Rather, the predictive motion mapping for the flexible device extends to all functionally equivalent structures, methods, and uses, such as those within the scope of the appended patent claims.

[0115]

[0124] The illustrations of the embodiments described herein are intended to provide a general understanding of the structures of the various embodiments. The illustrations are not intended to function as a complete description of all the elements and features of the disclosure described herein. Many other embodiments will be apparent to those of ordinary skill in the art upon consideration of the present disclosure. Other embodiments are utilized and derived from the present disclosure such that structural and logical substitutions and changes are made without departing from the scope of the present disclosure. Also, the illustrations are merely representative and may not be drawn to scale. Specific ratios within the illustrations may be exaggerated, while other ratios may be reduced. Accordingly, the present disclosure and the figures should be regarded as illustrative and not restrictive.

[0116]

[0125] One or more embodiments of the present disclosure may be referred to herein individually and / or collectively by the term "invention" merely for convenience and are not intended to limit the scope of this application to any particular invention or inventive concept. Further, although particular embodiments have been illustrated and described herein, it should be understood that any configuration derived therefrom that is designed to achieve the same or similar purpose may be substituted for the particular embodiments shown. The present disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above-described embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon review of this specification.

[0117]

[0126] The summary of the disclosure is provided in compliance with 37 C.F.R. §1.72(b) and is submitted with the understanding that it will not be used to limit the scope of the claims or their interpretation. Further, in the description of the embodiments for carrying out the foregoing invention, various features may be grouped together or described in a single embodiment for the purpose of briefly describing the present disclosure. The present disclosure should not be construed as reflecting an intention that the claimed embodiments require more functions than are expressly recited in each claim. Rather, the subject matter of the invention may be directed to less than all of the features of any of the disclosed embodiments, as reflected in the following claims. Accordingly, the following claims are incorporated into the description of the embodiments for carrying out the invention, and each claim stands on its own as defining separately claimed subject matter.

[0118]

[0127] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in this disclosure. Thus, the disclosed subject matter should be considered illustrative and not restrictive, and the appended claims are intended to cover all such modifications, extensions, and other embodiments that fall within the true spirit and scope of this disclosure. Accordingly, to the fullest extent permitted by law, the scope of this disclosure should be determined by the broadest permissible interpretation of the following claims and their equivalents, and should not be limited or restricted by the foregoing forms of implementing the invention.

Claims

1. 1. A controller comprising a memory for storing instructions and a processor for executing the instructions, the instructions, when executed by the processor, causing the controller to: obtaining at least one position of a distal end of an interventional medical device during an interventional medical procedure; determining motion at a proximal end of the interventional medical device during the interventional medical procedure; applying a first trained artificial intelligence to the motion at the proximal end of the interventional medical device and to the at least one location at the distal end of the interventional medical device; predicting a predicted movement along the interventional medical device toward the distal end of the interventional medical device during the interventional medical procedure based on applying the first trained artificial intelligence to the movement at the proximal end of the interventional medical device and the at least one location of the distal end of the interventional medical device; obtaining an image of the distal end of the interventional medical device from a medical imaging system after the motion at the proximal end of the interventional medical device is identified; comparing, from the image of the interventional medical device, an actual motion along the interventional medical device toward the distal end of the interventional medical device with the predicted motion along the interventional medical device toward the distal end of the interventional medical device; determining when the actual behavior deviates from the predicted behavior; controller.

2. the at least one position of the distal end of the interventional medical device is obtained from an image of the distal end of the interventional medical device from the medical imaging system; The controller of claim 1 , wherein the predicted motion along the interventional medical device toward the distal end of the interventional medical device is predicted for a plurality of positions along the interventional medical device toward the distal end of the interventional medical device.

3. the first trained artificial intelligence inputs at least one of a type of the interventional medical device, a type of the interventional medical procedure, an anatomical landmark, or at least one physical characteristic of a patient; The controller of claim 1 , wherein the predicted movement along the interventional medical device toward the distal end of the interventional medical device is predicted based on the input to the first trained artificial intelligence.

4. When executed by the processor, the instructions cause the controller to further: The controller of claim 1 , further comprising: generating an alarm when the actual performance deviates from the predicted performance.

5. When executed by the processor, the instructions cause the controller to further: The controller of claim 1 , further comprising: a controller configured to predict a predicted approximate estimated location of unintended motion along the interventional medical device outside the field of view of the medical imaging system based on the actual motion and the predicted motion and by applying a second trained artificial intelligence.

6. a motion detector for detecting motion at a proximal end of the interventional medical device; 1. A system for controlling an interventional medical device comprising: a memory storing instructions; and a controller comprising a processor executing the instructions, the instructions, when executed by the processor, causing the system to: obtaining at least one position of a distal end of an interventional medical device during an interventional medical procedure; determining the motion at the proximal end of the interventional medical device during the interventional medical procedure; applying a first trained artificial intelligence to the motion at the proximal end of the interventional medical device and to the at least one position at the distal end of the interventional medical device; predicting a predicted movement along the interventional medical device toward the distal end of the interventional medical device during the interventional medical procedure based on applying the first trained artificial intelligence to the movement at the proximal end of the interventional medical device and the at least one location of the distal end of the interventional medical device; causing the controller to acquire images of the distal end of the interventional medical device from a medical imaging system after the motion at the proximal end of the interventional medical device is identified; comparing, from the image of the interventional medical device, an actual motion along the interventional medical device toward the distal end of the interventional medical device with the predicted motion along the interventional medical device toward the distal end of the interventional medical device; determining when the actual behavior deviates from the predicted behavior; system.

7. The system further comprises the medical imaging system; the at least one position of the distal end of the interventional medical device is obtained from an image of the distal end of the interventional medical device from the medical imaging system; The system of claim 6 , wherein the images from the medical imaging system are segmented to identify the interventional medical device imaged by the medical imaging system.

8. When executed by the processor, the instructions further cause the system to: inputting at least one position of a distal end of the interventional medical device, identifying a motion at a proximal end of the interventional medical device, and detecting an actual motion along the interventional medical device toward the distal end of the interventional medical device resulting from the motion at the proximal end of the interventional medical device, in a plurality of training sessions for a plurality of interventional medical devices; predicting a predicted motion along the interventional medical device toward the distal end of the interventional medical device based on the at least one position of the distal end of the interventional medical device and the motion determined at the proximal end of the interventional medical device; determining a loss based on a difference between the predicted performance and the actual performance; establishing, by the first trained artificial intelligence, a relationship between the motion at the proximal end of the interventional medical device and the motion traveling through the interventional medical device near the distal end of the interventional medical device; The system of claim 6 , further comprising updating the first trained artificial intelligence based on each loss determined based on a difference between the predicted behavior and the actual behavior.

9. The system further includes an artificial intelligence controller that implements the first trained artificial intelligence and the second trained artificial intelligence, and when executed by the processor, the instructions further provide the system with: inputting ground truth information of an approximate estimated location of the interventional medical device outside the field of view of the medical imaging system during a plurality of training sessions; predicting a predicted approximate estimated location of the interventional medical device outside the field of view of the medical imaging system based on the predicted motion and the actual motion and by applying the second trained artificial intelligence; determining a loss based on a difference between the ground truth information of the approximate estimated location of the interventional medical device and the predicted approximate estimated location of the interventional medical device outside the field of view of the medical imaging system; The system of claim 6 , further comprising updating the second trained artificial intelligence based on each loss.

10. a robot that controls movement at the proximal end of the interventional medical device; an interface that outputs an alert based on the predicted movement along the interventional medical device toward the distal end of the interventional medical device within a field of view of the medical imaging system; and The system of claim 6 further comprising:

11. 1. A computer-implemented method for controlling an interventional medical device, comprising: a processor acquiring at least one position of a distal end of the interventional medical device from a medical imaging system during an interventional medical procedure; the processor determining a movement of a proximal end of the interventional medical device during the interventional medical procedure; applying, by the processor, a first trained artificial intelligence to the motion at the proximal end of the interventional medical device and to the at least one location at the distal end of the interventional medical device; predicting a predicted movement along the interventional medical device toward the distal end of the interventional medical device during the interventional medical procedure based on the processor applying the first trained artificial intelligence to the movement at the proximal end of the interventional medical device and the at least one location of the distal end of the interventional medical device; acquiring an image of the distal end of the interventional medical device from a medical imaging system after the processor has identified the motion at the proximal end of the interventional medical device; the processor comparing, from the image of the interventional medical device, actual motion along the interventional medical device toward the distal end of the interventional medical device to the predicted motion along the interventional medical device toward the distal end of the interventional medical device; the processor determining when the actual behavior deviates from the predicted behavior; 23. A computer-implemented method comprising:

12. The computer-implemented method of claim 11, further comprising the step of the processor segmenting the image of the interventional medical device to identify the interventional medical device imaged by the medical imaging system.

13. The method of claim 12, further comprising: predicting the predicted behavior accompanied by a confidence level of the predicted behavior; predicting, based on the actual motion, the predicted motion, and the confidence of the predicted motion, and by applying a second trained artificial intelligence, a predicted general estimated location of an unintended movement along the interventional medical device outside the field of view of the medical imaging system, and a predicted confidence of the predicted general estimated location; The computer-implemented method of claim 11 , further comprising:

14. The method of claim 13, further comprising: predicting the predicted motion based on at least one of a type of the interventional medical device, an anatomical structure of a patient during the interventional medical procedure, a positioning of the medical imaging system, or a physical characteristic of the interventional medical device; the processor outputting an alarm when the actual behavior deviates from the predicted behavior; The computer-implemented method of claim 11 , further comprising:

15. The step of the processor predicting a predicted approximate estimated location of an unintended behavior along the interventional medical device outside the field of view of the medical imaging system based on the actual motion and the predicted motion and by applying a second trained artificial intelligence; generating a representation of the predicted general estimated location of unintended movement along the interventional medical device outside the field of view of the medical imaging system; The computer-implemented method of claim 11 , further comprising:

Images