Flexible device length estimation from moving fluoroscopic images

Abstract

An apparatus (10) for determining an insertion length of an interventional instrument. The apparatus includes a processor (20) configured to receive imaging data (35) including an anatomical structure within a patient, identify a portion of an interventional instrument (12) inserted within the patient and disposed within the anatomical structure from the imaging data, and predict an insertion length of the interventional instrument based on the identified portion of the interventional instrument.

Description

Technical Field

[0001] The following generally relates to intravascular techniques, device selection techniques, artificial intelligence (AI) techniques, and related techniques. Background Art

[0002] Using an intravascular device of an appropriate length for each patient is important for ensuring the best outcome for the patient. As an example, depending on the application for which the catheter is used, the catheter length may vary for patients of different heights. For example, during central venous catheter (CVC) placement, an inappropriate catheter length may increase the risk of catheter migration or displacement (see, e.g., "Central Venous Catheter Intravascular Malpositioning: Causes, Prevention, Diagnosis, and Correction" by Roldan, C.J., and Paniagua, L. (2015) (The western journal of emergency medicine, 16(5), 658–664), https: / / doi.org / 10.5811 / westjem.2015.7.26248) and may require additional surgery to reposition the catheter and prevent vascular complications.

[0003] Even if an inappropriate catheter length is identified before the end of the procedure, additional surgical time is required to extract and insert a catheter of the appropriate length. A procedure using an intravascular robot may require even more time to replace the device because the old device will need to be removed from the robot and the new device will need to be inserted into the robot before being inserted into the patient's vasculature. The increased surgical time increases the risk of complications, and using multiple catheters increases waste and cost.

[0004] Certain improvements that overcome these and other problems are disclosed below. Summary of the Invention

[0005] In some embodiments disclosed herein, a system for determining an insertion length of an intervention device includes a processor in communication with a memory. The processor is configured to: receive imaging data including anatomic structures within a patient; identify a portion of the intervention device inserted into the patient and disposed within the anatomic structure based on the imaging data; obtain scaling information associated with the imaging data; and predict the insertion length of the intervention device based on the identified portion of the intervention device and the scaling information.

[0006] In some embodiments disclosed herein, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to: receive imaging data including an anatomical structure within a patient; identify a portion of an interventional device inserted into the patient and disposed within the anatomical structure based on the imaging data; obtain scaling information associated with the imaging data; and predict an insertion length of the interventional device based on the identified portion of the interventional device and the scaling information.

[0007] In some embodiments disclosed herein, a method of determining an insertion length of an interventional device. The method includes: receiving imaging data including an anatomical structure within a patient; identifying a portion of an interventional device inserted into the patient and disposed within the anatomical structure based on the imaging data; obtaining scaling information associated with the imaging data; and predicting an insertion length of the interventional device based on the identified portion of the interventional device and the scaling information.

[0008] One advantage is reducing delays and costs during endovascular surgery.

[0009] Another advantage is determining the insertion length of an endovascular device during endovascular surgery.

[0010] Another advantage is determining the insertion length of an endovascular device in real time during endovascular surgery.

[0011] Another advantage is determining the insertion length of an endovascular device during endovascular surgery based on medical imaging that is typically performed to provide image guidance to a surgeon during surgery.

[0012] Another advantage is using imaging in both determining the insertion length of an endovascular device for an endovascular surgery and a confidence value for the determined insertion length.

[0013] A given embodiment may not provide any of the foregoing advantages, provide one of the foregoing advantages, provide two of the foregoing advantages, provide more than two of the foregoing advantages, or provide all of the foregoing advantages, and / or may provide other advantages that will be apparent to those skilled in the art upon reading and understanding this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] This disclosure may take the form of various components and arrangements of components and various steps and arrangements of steps. The drawings are for the purpose of illustrating only the preferred embodiments and should not be construed as limiting this disclosure.

[0015] Figure 1 Schematically illustrates an endovascular device according to this disclosure.

[0016] Figure 2 Schematically illustrates a method of performing an endovascular treatment method using a Figure 1 device.

[0017] Figure 3 Schematically shows the operation of a model that receives as input images depicting a portion of a patient's anatomy and a portion of an associated interventional device within the patient, and outputs an estimated length of the interventional device inserted into the patient's anatomy.

[0018] Figure 4 Schematically shows by Figure 1 the visualization displayed by the device, where the field of view of the imaging device changes, resulting in a shift in the background, and where the device is inserted into the patient's anatomy, resulting in a shift in the device position.

[0019] Figure 5 Shows Figure 3 another embodiment of the operation of the NN. Detailed Description

[0020] During intravascular surgery in which a catheter or other interventional device is inserted into a blood vessel in a patient's vasculature, fluoroscopic imaging or another suitable imaging modality is typically used to visualize the insertion. Such procedures are sometimes referred to by terms such as image-guided therapy (IGT). Image guidance can be performed in real time, for example, as a time series of images, to provide a CINE view of the procedure. Image guidance can provide real-time visual guidance to the surgeon or other person performing the procedure regarding the current position of the interventional device (e.g., the tip of the interventional device) when the interventional device is inserted into the patient's body and regarding the surrounding vasculature and / or other tissues or organs.

[0021] In the embodiments disclosed herein, imaging such as for providing visual image guidance to an operator is also advantageously used to estimate and provide in real time the insertion length of an interventional device. The insertion length is the length of the interventional device currently inserted into the patient's body. The insertion length of the interventional device can include all or part of the entire length of the interventional device. Such an insertion length estimation based on a time series of images is challenging. The operator can adjust the position of the imaging field of view (FOV) to follow the tip of the interventional device as the interventional device advances through the body. Thus, the time series of images can include both the movement of the interventional imaging device and the movement of the background. These movements can typically be independent. In addition, the FOV will generally only contain the distal portion of the interventional device. Further still, the interventional images can be two-dimensional (2D) images, e.g., fluoroscopic images acquired using a flat detector panel, which further complicates the extraction of the length of the inserted portion of the device in three-dimensional (3D) space from the 2D images.

[0022] Recent work in machine learning and computer vision has explored methods for estimating features from moving objects by separating the target object from its background (see, e.g., "Learning the Depths of Moving People by Watching Frozen People" by Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T. Freeman (2019), CVPR). However, these methods have been developed for natural world data. For example, one application predicts the depth of an object (e.g., a person) in an image sequence with a moving or changing background (e.g., the camera moves with the person). In this application, the target depth and the background depth change at different rates. Since the camera moves with the person, the depth estimate of the person's arms and legs may only change to accommodate their walking, while the depth estimate of the background may change more drastically. However, natural world images contain more features and information than medical images. Some embodiments for estimating the insertion length of an interventional device adapt these techniques to this different task.

[0023] Reference Figure 1 , schematically shows system 10. System 10 may include, for example, an intravascular device, an endobronchial device, a surgical device (e.g., a needle), or any other suitable device. As Figure 1 shown, system 10 includes an interventional device or instrument 12 (e.g., a catheter, a guide wire, etc., schematically shown as a line in Figure 1 ), which is configured to be inserted into a portion of a patient's anatomy, such as into a blood vessel V containing a target (such as an occlusion or a clot, etc.). As in Figure 1As shown, the interventional device or instrument 12 is flexible such that it can follow the contour of the blood vessel V when inserted. In a typical endoluminal or intravascular procedure, a surgeon or other operator enters the target by creating an incision (not shown) and inserting the tip 14 of the interventional instrument 12 through the incision into the blood vessel V, and then pushing the interventional instrument 12 into and through the blood vessel V until the tip 14 reaches the target. The interventional instrument 12 is typically radiopaque to at least the extent that it is visible (potentially with low contrast) in X-ray imaging. The interventional instrument 12 optionally includes a highly radiopaque tip element 15 located at its tip 14 such that the tip 14 of the interventional instrument 12 is more easily imaged in fluoroscopic imaging. For example, the tip element 15 located at the tip element 14 of the interventional instrument 12 can be a coating of radiopaque material disposed on the tip 14, or can include an attached radiopaque ring 15 made of, for example, platinum or nitinol wire, which is metallurgically (e.g., by welding) bonded to the tip 14 of the interventional instrument 12. These are merely illustrative examples.

[0024] In an exemplary embodiment, a clinician controls the movement of the interventional instrument 12 through the blood vessel V; however, the movement of the interventional instrument 12 can also be robotically controlled. Figure 1 Also shown is a robot 16 (schematically shown as a box in Figure 1 ), which is operatively connected to the proximal end of the interventional instrument 12, i.e., to the end opposite the tip 14. The robot 16 (and more generally the proximal portion of the interventional instrument 12) is located outside the patient and more specifically outside the blood vessel V. As the interventional instrument 12 is pushed into the blood vessel V (manually or automatically), the length of the interventional instrument 12 disposed within the blood vessel V increases. Optionally, the robot 16 is configured to control the movement of the interventional instrument 12 into and through the blood vessel V. The robotic control can be performed by a clinician using a controller such as a joystick or a mouse click on a user interface, or can be automatically performed using an autonomous control system capable of manipulating the robot 16.

[0025] Figure 1Also shown is a hardware processing device 18, such as a workstation computer, a smart tablet, or more generally a computer, which can be used to control the robot 16 to automatically perform the insertion and movement of the intervention device 12 through the blood vessel V. The processing device 18 can also include a server computer or multiple server computers, for example, interconnected to form a server cluster, cloud computing resources, etc., to perform more complex computing tasks. The electronic processing device 18 includes typical components, such as a hardware processor 20 (e.g., a microprocessor), at least one user input device (e.g., a mouse, a keyboard, a trackball, and / or others) 22, and a display device 24 (e.g., an LCD monitor, a plasma monitor, a cathode ray tube monitor, and / or others). In some embodiments, the display device 24 can be a component separate from the processing device 18, or can include two or more display devices.

[0026] The processor 20 is operably connected to one or more non-transitory storage media 26. As a non-limiting illustrative example, the non-transitory storage media 26 can include one or more of the following: magnetic disks, RAID, or other magnetic storage media; solid state drives, flash drives, electrically erasable read-only memories (EEROMs), or other electronic memories; optical discs or other optical storage devices; various combinations thereof; or others; and can be, for example, a network storage device, an internal hard disk drive of the electronic processing device 18, various combinations thereof; or others. It should be understood that any reference herein to one or more non-transitory media 26 will be broadly construed to cover a single medium or multiple media of the same or different types. Similarly, the processor 20 can be embodied as a single processor or two or more processors. The non-transitory storage media 26 stores instructions executable by at least one processor 20. The instructions include instructions for generating a visualization of a graphical user interface (GUI) 28 for display on the display device 24.

[0027] Figure 1 Also schematically shown is an imaging device 30, which is configured to acquire a single-shot image and / or a time series of images or imaging frames 35 of the position or movement of the intervention device 12 (e.g., the distal portion of the device 12 including the radiopaque tip 14). In the illustrative example, the imaging device 30 is a fluoroscopic imaging device (e.g., an X-ray imaging device, a C-arm imaging device, a CT scanner, etc.), and the intervention device 12 is visible under fluoroscopic imaging. In some embodiments, the fluoroscopic imaging is real-time imaging, for example, in some non-limiting illustrative embodiments, images are acquired at a frame rate of 15 - 60 frames per second (i.e., 15 - 60 fps). The imaging device 30 can communicate with at least one processor 20 of the processing device 18. As Figure 1As shown, the imaging device 30 includes an X-ray imaging device, which includes an X-ray source 32 and an X-ray detector 34, such as a C-arm imaging device; however, it should be understood that any suitable imaging device can be used, such as ultrasound (US), computed tomography (CT), flat-panel X-ray or fluoroscopy, magnetic resonance imaging (MRI), or any other suitable imaging device. It should be noted that Figure 1 schematically illustrates the X-ray source 32 and the X-ray detector 34. In practice, the field of view of the imaging device 30 should be large enough to at least enclose the distal portion of the interventional device 12, which may include the tip 14 of the interventional device 12 and the surrounding vasculature and / or anatomical structure. The image 35 can be stored in the non-transitory storage medium 26.

[0028] At least one processor 20 is configured to execute the process 100 as described above, which can be a vascular diagnostic method, a vascular treatment method, an endobronchial diagnostic method, an endobronchial treatment method, a surgical method, etc. The non-transitory storage medium 26 stores instructions that can be read and executed by the at least one processor 20 to perform the disclosed operations, the disclosed operations including performing, for example, a vascular treatment method or the process 100. In some examples, the method 100 can be performed at least in part by cloud processing.

[0029] Referring Figure 2 and continuing to refer Figure 1 to, an illustrative embodiment of the method 100 is shown as a flowchart. To begin the method 100, the interventional device 12 is inserted into the blood vessel V using the robot 16 or manually and pushed into the vasculature.

[0030] At operation 102, the imaging device 30 acquires a time series of images 35 (or imaging frames) of the patient. The received time series of images 35 can include two-dimensional (2D) imaging data that includes a portion of the interventional device 12 and a portion of the patient's anatomical structure in which that portion of the interventional device is disposed (e.g., the blood vessel V). The imaging operation 102 can be performed during the interventional procedure to provide visual guidance to the physician as the physician manipulates the proximal end of the interventional device 12 to push the device 12 through the vasculature toward a clot or other target within the patient's body. The imaging device 30 transmits the time series of images 35 to the processing device 18.

[0031] At operation 104, the processing device 18 identifies a portion of the interventional device 12 in the sequence of images 35. In an embodiment, all or part of this portion of the interventional device is invisible or undetectable by a user (e.g., a physician) of the system 10 in the image sequence. The identification can be performed by automatic segmentation based on prior knowledge of the expected appearance of the interventional device 12 in the image, such as the expected width of the device image based on its known diameter and imaging characteristics, the expected gray intensity range of the image pixels representing the interventional device 12 in the image (e.g., in the case of X-ray imaging, based on the radiodensity of the interventional device 12 on the Hounsfield scale), prior knowledge of the maximum bending radius of the interventional device 12, and / or others.

[0032] At operation 106, the processing device 18 obtains scaling information indicating the scaling associated with the image 35. The processing device can determine the length of the interventional device 12 inserted into the patient based on the image 35 and the scaling information. In some embodiments, the received scaling information includes one or more of the amount of movement of components of the system 10, patient size information, a three-dimensional (3D) image of the portion of the patient's anatomy where the interventional device 12 is disposed, and size information of the interventional device 12 (e.g., length, size (e.g., 6Fr), distance between the tip element 15 and the tip 14, flexibility, etc.).

[0033] At operation 108, the processing device 18 determines, estimates, or predicts the insertion length of the interventional device 12 based on the time series of the images 35. In some embodiments, only a portion of the interventional device is currently inserted into the patient's anatomy, and the insertion length is the length of the currently inserted portion of the interventional device. In an embodiment, the insertion length can include all or part of the entire length of the interventional device. In some embodiments, the insertion length can be determined based on features of the portion of the interventional device extracted from the image (e.g., shape, size, position, components on the device, etc.) and features of the surrounding anatomy extracted from the image (e.g., landmarks, shape, size, in-vivo position, etc.), such as by image feature extraction techniques known in the art. In some embodiments, the insertion length can also be determined based on the scaling information of operation 106. In an exemplary embodiment, the processing device 18 can calculate the length of the portion of the interventional device in the image based on the extracted portion and the features of the surrounding anatomy, and then apply the scaling information to the calculated length of the portion of the interventional device in the image to calculate the insertion length of the interventional device.

[0034] In some embodiments, the determination of the insertion length of the interventional device 12 can be performed by implementing a model 36, such as a machine learning (ML) model based on a handcrafted feature extractor. For example, the model can include relevant image parameters or features extracted by a feature extractor applied to Gaussian mixture models, expectation maximization, hidden Markov models, etc. In some embodiments, the model can be based on features learned using a neural network (NN). Briefly referring to Figure 3 , in some embodiments, the processing device 18 can perform the insertion length determination by applying a model 36 of an artificial neural network (ANN) 36 configured to be stored in the non-transitory storage medium 26 of the processing device 18. The ANN 36 may have been previously trained based on historical data (such as historical imaging data and patient data (e.g., intravascular imaging data and intravascular patient data)) to determine the insertion length of the interventional device 12 based on the current portion of the interventional device present in the current image and the current portion of the anatomical structure present in the current image. In some embodiments, the historical data can include historical scaling information, and the ANN has been trained to also determine the insertion length based on the current scaling information.

[0035] As Figure 3 schematically shown, the ANN 36 can receive a time series of images 35 and other data (indicated as element 38) as inputs, including, for example, patient health information (e.g., patient height, patient age, etc.), preoperative or intraoperative three-dimensional (3D) images of the patient (including portions of the interventional device 12), C-arm movement amount, patient table movement amount, or any other information that allows the estimated device length from the image 35 to be scaled to a metric length. Then, the ANN 36 can predict and output the metric length of the device inserted into the patient's anatomical structure (indicated as element 39) based on the input data. In some embodiments, a time series of images 35 can be utilized to train the ANN 36. The training of the ANN 36 can include adjusting ANN parameters including model weights and biases using training data (e.g., image data, scaling data, etc. from previous surgeries) such that the trained ANN 36 accurately predicts the expected output data based on new input data.

[0036] Returning to Figure 2, in some embodiments, the processing device 18 (operation 108) determines the insertion length of the interventional device 12 based on an analysis of the portion of the interventional device 12 present in the image 35 and one or more anatomical landmarks present in the image 35. In some embodiments, the processing device 18 applies an artificial neural network (ANN) 36 that is trained to determine such insertion length using image features extracted from the image, the image features including features of the portion of the interventional device 12 present in the image 35 and one or more anatomical landmarks present in the image 35. In some embodiments, there may not be a continuous stream of fluoroscopic images 35 from the entry site to the current position of the interventional device 12 within the anatomical structure. In some cases, navigation of the interventional device 12 near the entry site is not performed under fluoroscopy, and fluoroscopy is only initiated near more complex vasculature. In other cases, fluoroscopy may be used near the entry site but not in subsequent segments and then restarted near more complex vasculature. In such cases, data is missing from the acquired images 35, and the ANN 36 must infer the insertion length of the interventional device 12 from background features that inform which portion of the anatomical structure is currently being imaged. For example, image features in the background along with patient height can allow the processing device 18 to estimate a metric length of the insertion length of the interventional device 12 even if a previous fluoroscopic sequence is not available.

[0037] In some embodiments, at least one image of the imaging data 35 includes at least two images depicting different views of the portion of the interventional device 12 present in the image and one or more anatomical landmarks. In this embodiment, if multiple C-arm views from the same time are available (e.g., data acquired from a biplane system), the processing device 18 uses data from the multiple views to compute a more accurate estimate of the length of the interventional device 12 currently inserted into the patient because the data will provide more information about the interventional device 12 and the background and thus increase the confidence of the prediction of the processing device 18 (e.g., reduce the ambiguity of foreshortening in fluoroscopy). The multiple views can be input into the ANN 36 as separate input channels or into a siamese network architecture as separate inputs, where parallel convolutional layers process the multiple views separately in the early layers of the ANN 36 and combine the ANN 36 weights in a later layer to provide a combined output.

[0038] In some embodiments, the processing device 18 (operation 108) tracks the insertion device length estimate according to a previous image (or imaging frame) 35 to generate a consistent output as the interventional instrument 12 is inserted into the patient. Such tracking can be performed as a post-processing step to generate a smooth output of the estimated device length, for example, by outputting the average of a set of most recent device length predictions from consecutive imaging frames 35. Alternatively, the ANN 36 can use its most recent output or a set of most recent outputs as input to predict the subsequent length of the interventional instrument 12 inserted into the patient (as indicated by the dashed arrow in Figure 3 ).

[0039] In some embodiments, the processing device 18 (operation 108) determines the length of the interventional instrument 12 currently inserted into the patient as the cumulative incremental change in the insertion length in consecutive images of the time series of images 35. In some embodiments, the ANN 36 is configured to determine such insertion length as the cumulative of such incremental changes in consecutive images of the time series of images 35. To this end, the processing device 18 can be configured to determine the incremental change in the insertion device length between images of a pair of images of the time series of images 35 by inputting each image pair or features extracted therefrom into the ANN 36 trained to output the incremental change in the insertion length of the instrument, and the processing device 18 sums the determined incremental changes to determine the current insertion length of the instrument. In some embodiments, determining the insertion length as the cumulative of the incremental changes in the insertion length includes inputting the time series of images 35 (or features extracted therefrom) into the ANN 36 (implemented as a temporal ANN) that is trained to output the insertion length based on the time series of input images or features extracted therefrom.

[0040] Reference Figure 4 , an illustrative example of the time series of images 35 is shown by way of an illustrative image at time t = t0 and a later illustrative image at time t = t n . In each image, the depiction of the interventional instrument is indicated and labeled as instrument depiction 12I. Figure 4 Shows that between time t = t0 and time t = t n , the interventional instrument has moved (as seen by comparing its image 12I in the two images), and the background has moved because the operator has moved the imaging device 30 to move the imaging field of view (FOV) to capture the branch region. This is indicated in Figure 4 by the illustration “shift in device” 50 and “shift in background” 52.

[0041] Reference Figure 5, in some embodiments, determining the current insertion length of the instrument includes, for each pair of consecutive images in the time series of images 35, calculating the optical flow field 56 between the images of the pair, identifying the portion of the intervention instrument 12 depicted in each image of the pair (represented as "device mask" 58 in Figure 5 ), and inputting the optical flow field 56 and the identified portion 58 of the intervention instrument 12 depicted in each image of the pair (optionally, together with additional information 38) into the ANN 36, which is trained to determine the insertion device length 39 from the input. In this method, the optical flow field 56 captures the background movement or shift 52 that occurs during the time interval between the images of the pair (see Figure 4 ), while the identified portion 58 of the intervention instrument 12 depicted in each image captures the device movement or shift 50 that occurs during that time interval. In some embodiments, calculating the optical flow field 56 includes masking the identified portion of the intervention instrument 12 depicted in each image of the pair before calculating the optical flow field. This device masking can ensure that the optical flow field 56 represents only the background movement or shift 52 and does not have a contribution from the typically independent device movement 50. However, since the fraction of the total area of each image occupied by the (usually thin) intervention instrument 12 is small, in some other embodiments, this masking before calculating the optical flow field 56 can be omitted. In some embodiments, the images of the pair can be directly input into the ANN 36 together with the optical flow field 56 or the identified portion 58 of the intervention instrument 12, allowing the ANN 36 to automatically learn the relevant features that lead to an accurate estimate of the insertion device length 39.

[0042] Return reference Figure 2, at operation 110, the processing device 18 generates a confidence value for the determined insertion length of the interventional device. In some embodiments, the confidence value can be directly estimated by the ANN 36 as an additional output. During the training of the ANN 36, the confidence (c) output can be compared with the error (e) in the length estimation of the interventional device 12 (e.g., c = 1 / e). An instance with a lower error means a higher confidence, while an instance with a higher error means a lower confidence. Alternatively, a dropout layer in the ANN 36 can be used to calculate the confidence value. Dropout randomly discards the outputs of a specified number of nodes in the ANN 36, thus generating slightly different outputs for the same input during multiple inference runs. The mean and variance can be calculated from the multiple outputs, and as described above, a smaller variance indicates a higher confidence (consistent outputs), while a larger variance indicates a lower confidence (inconsistent outputs). These or other confidence estimation methods learn to associate lower confidence with features that tend to generate higher errors. For example, the ambiguity caused by the 2D nature of the fluoroscopic image (such as foreshortening (i.e., out-of-plane movement)) may result in higher errors. Similarly, the movement and appearance of background features (e.g., bony landmarks) away from the center of the image may be distorted due to the parallax effect. The parallax effect occurs because the X-ray source 32 is smaller than the X-ray detector 34, which means that away from the center of the image, the X-ray beam arrives at the detector at an inclined angle. These distortions may result in higher errors.

[0043] At operation 112, the processing device 18 outputs the determined insertion length, for example, on a display device 24 in communication with the processing device 18. In some embodiments, a visualization 38 of the length of the interventional device 12 is generated and displayed on the display device 24. The estimated length of the interventional device 12 is displayed relative to the portion of the patient's anatomy into which the interventional device 12 is inserted.

[0044] In some embodiments, the trained ANN 36 can be configured to take as input a sequence of fluoroscopic images 35 and other relevant information and compute an estimated length of the interventional device 12 (e.g., a guide wire) inserted into a patient. The ANN 36 can also be configured to compute this estimate by estimating the amount of movement in the background image and the amount of movement in the interventional device 12 to estimate the total movement, and scaling the estimate by the size of landmarks visible in the image background and / or by the thickness of the interventional device 12. This scaling allows the ANN 36 to estimate the metric length of the device inserted into the patient. The estimate can then be used to evaluate the length of a subsequent interventional device 12 to be inserted into the patient. The estimate can be used for downstream estimations, including but not limited to estimating the length of an interventional device 12 to be subsequently inserted into the patient. Other downstream estimations can include identifying anomalies when using the robot 16 to insert the interventional device 12. For example, the robot 16 can track the length of the interventional device 12 inserted into the patient at the entry site and can compare this known length to the length of the interventional device 12 inserted into the patient. In the case of a mismatch, the robot 16 can warn the user that the interventional device 12 may be buckling, for example, outside of the imaging field of view, and the user can take relevant action to address the buckling.

[0045] To train the ANN 36, retrospective data can be obtained. The retrospective data can be obtained from a large collection of historical surgeries, including (i) sequences of fluoroscopic images that include devices that may move, backgrounds that may move, and devices and backgrounds that both move, and (ii) other information available during the surgery regarding the patient and / or the surgery, including but not limited to C-arm movement, patient table movement, patient health information (e.g., patient height, patient age, etc.), pre-operative or intra-operative 3D images, or any information that allows scaling to a metric length for an estimated length based on the fluoroscopic image.

[0046] Next, the true length of the interventional device 12 inserted into the patient can be obtained as follows: For example, (i) a shape sensing device (e.g., Fiber Optic RealSense or FORS device) from which shape and / or other information from the interventional device 12 can be used to evaluate which parts of the interventional device 12 are inside the patient and which parts are outside the patient; (ii) an interventional device 12 with an electromagnetic (EM) tracking tip, if the position where the tip enters the patient's body is known and the tip is continuously tracked once the interventional device 12 is in the patient's body, then the length of the interventional device 12 inside the patient can be evaluated; (iii) a robot 16, where the length of the interventional device 12 that the robot 16 has pushed into the patient is known, or any manually inserted interventional device 12, where after the interventional device 12 is retrieved from the patient's body, the interventional device 12 can be observed to evaluate which part of the interventional device 12 was inserted into the patient and that part can be measured; alternatively, before the interventional device 12 is retrieved from the patient's body, the external part of the interventional device 12 closest to the entry site can be marked to evaluate which part of the interventional device 12 was inserted into the patient and that part can be measured.

[0047] As an example, in one implementation, the optical flow can be calculated between image pairs in a sequence of fluoroscopic images 35. However, since the interventional device 12 moves independently of the background, it can be segmented out from the optical flow calculation. This sequence of optical flow fields with a change mask identifying the interventional device 12 can be input into the ANN 36. The ANN 36 can be any architecture capable of processing temporal data, including but not limited to temporal convolutional networks (TCNs), recurrent neural networks (RNNs), transformer networks, etc. The ANN 36 uses features from these inputs to evaluate, scaled, how much of the interventional device 12 has been inserted.

[0048] Additional patient information allows the ANN 36 to scale the estimate to generate a metric length. The additional information 38 is processed differently depending on its type. For example, PHI (e.g., patient height, age, etc.), C-arm or table movement amounts, or other numerical information can be concatenated into the feature vector (e.g., a 1D vector or a linear layer of the first few layers before the output layer), as Figure 5 shown by the shaded circles in. For example, an ANN 36 trained with fluoroscopic images and patient height can learn to associate the estimated distance with landmarks (e.g., vertebrae) seen in the background fluoroscopic image. Another example of additional patient information is 3D image data. The 3D image data can be incorporated through registration. If the 2D fluoroscopy and 3D image are registered, then the scale of the anatomical structures visible in the fluoroscopic image is known.

[0049] The ANN 36 can be trained by calculating the error (e) between the network output and the true inserted device length. The error can be calculated using any loss function, including but not limited to L1 norm, L2 norm, negative log likelihood, etc. During training, the value of the loss function is typically minimized, and training terminates when the value of the loss function meets a stopping criterion. Sometimes, training terminates when the value of the loss function meets one or more of multiple criteria. Various algorithms have been developed to solve the loss minimization problem, including but not limited to Stochastic Gradient Descent "SGD", Batch Gradient Descent, Mini-Batch Gradient, Adam, etc. These algorithms use the chain rule to calculate the derivative of the loss function with respect to the model parameters. This process is called backpropagation because the derivatives are calculated starting from the last ANN layer or output layer and moving towards the first ANN layer or input layer. These derivatives inform the algorithm how the model parameters must be adjusted to minimize the loss function. If the training process is successful, the trained ANN 36 accurately predicts the expected output data based on new input data.

[0050] The output metric length of the inserted intervention device 12 can be visualized on a screen or communicated to the user in another way (e.g., audio feedback). This output notifies the subsequent steps. For example, the estimated metric length of a guide wire can help determine the length of the catheter inserted over the guide wire, as described above.

[0051] In some embodiments, synthetic data can be used for training, where various attributes that control X-ray image generation can be controlled, and thus allow the trained ANN 36 to be more robust. For example, the parallax effect and the resulting device distortion can be simulated in order to train an ANN 36 that is robust to distortions in devices away from the center of the image.

[0052] The present disclosure has been described with reference to preferred embodiments. Modifications and changes can be made upon reading and understanding the foregoing detailed description. The exemplary embodiments are intended to include all such modifications and changes as long as they fall within the scope of the claims or their equivalents.

Claims

1. A system (10) for determining the insertion length of an interventional device (12), the system comprising: a processor in communication with a memory, the processor being configured to: receive imaging data (35) comprising an anatomical structure within a patient; identify, based on the imaging data, a portion of the interventional device (12) inserted into the patient and disposed within the anatomical structure; obtain scaling information associated with the imaging data; and predict the insertion length of the interventional device based on the identified portion of the interventional device and the scaling information.

2. The system (10) according to claim 1, wherein, The received imaging data (35) comprises a time series of images, and the processor (20) is further configured to predict the insertion length of the interventional device as a cumulative sum of incremental changes in the insertion length of the interventional device between consecutive images of the time series of images.

3. The system (10) according to claim 1, wherein, The processor is further configured to: for each pair of consecutive images of the time series of images: calculate an optical flow field between the images of the image pair; identify the portion of the interventional device (12) in each image of the image pair; and apply a model (36) that is trained to predict the insertion length based on the optical flow field, the identified portion of the interventional device in each image of the image pair, and the scaling information.

4. The system (10) according to claim 3, wherein, The processor is configured to mask the identified portion of the interventional device (12) in each image of the image pair prior to calculating the optical flow field.

5. The system (10) according to claim 2, wherein, The processor is further configured to: apply a model that is trained to predict the incremental change in the insertion length based on images of image pairs of the time series of images or features extracted from images of image pairs of the time series of images.

6. The system (10) according to claim 5, wherein, The processor is further configured to: sum the predicted incremental changes to determine the insertion length.

7. The system (10) according to claim 1, wherein, The processor is configured to: predict the insertion length of the interventional device based on the identified portion of the interventional device disposed within the anatomical structure, the scaling information, and one or more anatomical landmarks of the anatomical structure in the imaging data.

8. The system (10) according to claim 7, wherein, The imaging data comprises at least two images that depict different views of the portion of the interventional device (12) and the one or more anatomical landmarks; and the processor is configured to predict the insertion length of the interventional device based on the at least two images depicting the different views.

9. The system (10) according to any one of claims 1-8, wherein, The scaling information comprises a dimensional scale associated with the portion of the interventional device (12).

10. The system (10) according to any one of claims 1 - 8, wherein, The scaling information comprises one or more of: a movement amount of a component of an imaging device that acquired the imaging data, patient size information, a three-dimensional (3D) image of a portion of the patient's anatomical structure in which the interventional device is disposed, and information about the size of a related interventional device to be inserted into the patient's anatomical structure.

11. The system (10) according to any one of claims 1-10, wherein, The processor (20) is configured to output the predicted insertion length on a display device (24).

12. The system (10) according to claim 11, wherein, The processor (20) is further configured to: generate a visualization (38) of the length of the interventional device; and Display a visualization of the length of the generated interventional instrument on the display device (24).

13. The system (10) according to any one of claims 1-12, wherein, The processor (20) is configured to: Generate a confidence value for the insertion length.

14. The system (10) according to any one of claims 1-13, wherein, The processor (20) is further configured to: Determine the length based on features in the background of the images in the imaging data (35).

15. The system (10) according to any one of claims 1-14, wherein, The processor is configured to apply a machine learning model that is trained to predict the insertion length of the interventional instrument based on the identified portion of the interventional instrument disposed within the anatomical structure and the scaling information.

16. The system (10) according to any one of claims 1 - 15, wherein, The imaging data includes two-dimensional (2D) imaging data of the portion of the interventional instrument (12) and a portion of the patient's anatomical structure.

17. The system (10) according to any one of claims 1-16, further comprising: An imaging device (30) configured to acquire the imaging data; Wherein the imaging device communicates with the processor (20).

18. A non-transitory computer-readable medium (26) storing a computer program including instructions that, when executed by a processor (20), cause the processor to: Receive imaging data (35) including an anatomical structure within a patient; Identify, based on the imaging data, a portion of the interventional instrument inserted into the patient and disposed within the anatomical structure; Obtain scaling information associated with the imaging data; And Predict the insertion length of the interventional instrument based on the identified portion of the interventional instrument and the scaling information.

19. The non-transitory computer-readable medium (26) according to claim 18, wherein, The instructions, when executed by the processor, further cause the processor to: For each pair of consecutive images in a time series of images: Calculate an optical flow field between the images of the image pair; Identify the portion of the interventional instrument (12) in each image of the image pair; And Apply a model (36) that is trained to predict the insertion length based on the optical flow field and the identified portion of the interventional instrument in each image of the image pair.

20. A method (100) for determining the insertion length of an interventional instrument (12), the method comprising: Receiving imaging data (35) including an anatomical structure within a patient; Identifying, based on the imaging data, a portion of the interventional instrument (12) inserted into the patient and disposed within the anatomical structure; Obtaining scaling information associated with the imaging data; And Predicting the insertion length of the interventional instrument based on the identified portion of the interventional instrument and the scaling information.

Images