System and method for determining a procedure for retrieving fragment of endovascular device

Abstract

A system is provided for determining a workflow with regard to a fragment of a fractured endovascular device in a vasculature of a patient. The system includes a processor configured to receive at least one medical image of the vasculature, including the fragment of the endovascular device; extract at least one image feature in the at least one medical image; obtain a plurality of types of procedures for retrieval of an object from the vasculature; and determine a type of procedure, from the plurality of types of procedures, to retrieve the fragment from the vasculature based on the at least one image feature.

Description

SYSTEM AND METHOD FOR DETERMINING A PROCEDURE FOR RETRIEVINGFRAGMENT OF ENDOVASCULAR DEVICEBACKGROUND

[0001] According to the Food and Drug Administration (FDA) Manufacturer and User Facility Device Experience (MAUDE) database, neurovascular recanalization devices may fracture during interventional procedures due to excessive force that may be applied during advancement or retraction of the device. Examples of interventional devices, such as neurovascular recanalization devices, susceptible to breakage include aspiration catheters, guide catheters, stent retrievers, and others. Factors that affect the risk of device fracture include tortuosity of the vasculature, original occlusion site, device characteristics, and physician experience. FIGs. 1 A and IB show examples of tortuous vascular anatomy known to increase the risk of device fracture. In particular, FIG. 1 A shows a proximal loop 11 present in a left vertebral artery 10, and FIG. IB shows artery stenosis 22 present in a cavernous segment of a left internal carotid artery 20.

[0002] Once an interventional device has fractured within the vasculature of a patient, a determination must be made as to whether the fragments of the fractured interventional device need to be (or can be) retrieved and, if so, what procedures and / or device should be used to retrieve them. Factors that affect the success of fragment retrieval include the nature of the fracture site, fragment size, vascular anatomy, auxiliary retrieval devices, and physician experience and preference. Depending on these factors, different approaches may be taken by different physicians. For example, using snares is currently the most common retrieval technique. Other common device retrieval methods include leveraging aspiration or guide catheters, angioplasty balloons, or stent retrievers, as well as surgical procedures for more difficult cases. Surgical procedures are typically the last resort after failure of the interventional approaches, but only when the fragments are accessible (e.g., in the common carotid artery). When the fragments are inaccessible or after multiple trial and error attempts using various approaches, the physician may decide to leave the fragments in the vasculature. However, such a decision is not preferable since leaving the fragments in the vasculature may lead to increased risks of fragment migration, distal embolization and even stroke in the future, and thus demandsregular post-procedure monitoring of the patient. The monitoring itself may be harmful to the patient, e.g., due to contrast injections, X-ray exposure, time requirements, and cost.

[0003] What is needed is a decision support algorithm based on machine learning models, which predicts the most proper retrieval strategies, as well as devices to be used in the retrievals based on the intra-procedural angiography images and available auxiliary retrieval devices.SUMMARY

[0004] According to a representative embodiment, a system is provided for determining workflow with regard to a fragment of a fractured endovascular device in a vasculature of a patient. The system includes a processor in communication with memory. The processor configured to receive at least one medical image of the vasculature, including the fragment of the endovascular device; extract at least one image feature from the at least one medical image; obtain a plurality of types of procedures for retrieval of an object from the vasculature; and determine a type of procedure, from the plurality of types of procedures, to retrieve the fragment from the vasculature based on the at least one image feature. To determine the type of procedure, the processor may be configured to apply a machine learning model configured to receive input data, including the at least one image feature; predict, for each of the plurality of types of procedures, a probability of a successful outcome of retrieval of the fragment from the vasculature based on the input data; and determine the type of procedure to retrieve the fragment from the vasculature based on the predicted probability for each of the plurality of types of procedures.

[0005] According to another representative embodiment, a method is provided for determining a workflow with regard to a fragment of fractured endovascular device in a vasculature of a patient. The method includes receiving at least one medical image of the vasculature, including the fragment of the endovascular device; extracting at least one image feature from the at least one medical image; and based on the at least one image feature, determining a type of procedure, from a plurality of types of procedures, to retrieve the fragment from the vasculature. The method may further include applying a machine learning model comprising receiving input data, including the at least one image feature; predicting, for each of the plurality of types of procedures, a probability of a successful outcome of retrieval of thefragment from the vasculature based on the input data; and determining the type of procedure to retrieve the fragment from the vasculature based on the predicted probability for each of the plurality of types of procedures.

[0006] According to another representative embodiment, a non-transitory computer readable medium having stored instructions for determining a workflow with regard to a fragment of fractured endovascular device in a vasculature of a patient. When executed by a processor, the instructions cause the processor to receive at least one medical image of the vasculature, including the fragment of the endovascular device; extract at least one image feature from the at least one medical image; obtain a plurality of types of procedures for retrieval of an object from the vasculature; and determine a type of procedure, from the plurality of types of procedures, to retrieve the fragment from the vasculature based on the at least one image feature. To determine the type of procedure, the processor may be configured to apply a machine learning model configured to receive input data, including the at least one image feature; predict, for each of the plurality of types of procedures, a probability of a successful outcome of retrieval of the fragment from the vasculature based on the input data; and determine the type of procedure to retrieve the fragment from the vasculature based on the predicted probability for each of the plurality of types of procedures.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.

[0008] FIG. 1A shows an example of tortuous vascular anatomy including a proximal loop in a left vertebral artery, which increases risk of interventional device fracture.

[0009] FIG. IB shows an example of tortuous vascular anatomy including artery stenosis in a cavernous segment of a left internal carotid artery, which increases risk of interventional device fracture.

[0010] FIG. 2 is a simplified block diagram of a system for determining a workflow with regard to retrieving a fragment of a fractured endovascular device in a vasculature of a patient,according to a representative embodiment.

[0011] FIG. 3 is a schematic diagram showing how embedding space of the trained machine learning model (e.g., neural network) is leveraged to make predictions at inference, according to a representative embodiment.

[0012] FIG. 4A is a flow diagram of a method for determining a workflow with regard to retrieving a fragment of a fractured endovascular device in a vasculature of a patient, according to a representative embodiment.

[0013] FIG. 4B is a continuation of the flow diagram of the method for determining the workflow with regard to retrieving a fragment of a fractured endovascular device in a vasculature of a patient, according to a representative embodiment.DETAILED DESCRIPTION

[0014] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only, and is not intended to be limiting. The defined terms are in addition to the technical and scientific meanings of the defined terms as commonly understood and accepted in the technical field of the present teachings.

[0015] It will be understood that, although the terms first, second, third etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0016] The terminology used herein is for purposes of describing particular embodimentsonly, and is not intended to be limiting. As used in the specification and appended claims, the singular forms of terms “a,” “an” and “the” are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms “comprises,” and / or “comprising,” and / or similar terms when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0017] Unless otherwise noted, when an element or component is said to be “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0018] As used in the specification and appended claims, and in addition to their ordinary meanings, the term “about” and “approximately” mean to with acceptable limits or degree. For example, “approximately 2 MHz” means one of ordinary skill in the art would consider the signal to be 2 MHz within reasonable measure. Also, as used in the specification and appended claims, in addition to its ordinary meaning, the term “substantially” means within acceptable limits or degree. For example, the term “substantially simultaneously” means one of ordinary skill in the art would consider occurrence at the same time.

[0019] In view of the foregoing, the present disclosure, through one or more of its various aspects, embodiments and / or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below. For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. However, other embodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Moreover, descriptions of well-knownapparatuses and methods may be omitted so as to not obscure the description of the example embodiments. Such methods and apparatuses are within the scope of the present disclosure.

[0020] Generally, the various embodiments provide a process that can estimate interventional risks and benefits of retrieving fragments of endovascular devices that have fractured in the vasculature of a subject (patient). The risks and benefits may be estimated based on information extracted from intra-operative medical images (e.g., fluoroscopy images), as well as device information about the fractured endovascular device (e.g., size, stiffness) and / or patient information about the patient, e.g., from electronic hospital records (EHRs), patient health information (PHI). Further, based on outcome information associated with retrospective data, the process may suggest the type of procedure (e.g., interventional procedure, surgical procedure, no procedure), and may also suggest the preferred interventional or retrieval device(s) in the case of interventional procedure, which are predicted to result in a higher retrieval success probability. In example embodiments, the process may include one or more of a machine learning process, a simulation process, an image processing process, a relational database process, a rules-based process, etc. to predict the type of procedure and preferred device based on the described information.

[0021] FIG. 2 is a simplified block diagram of a system for determining a workflow for retrieving a fragment of an endovascular device that has fractured in vasculature of a patient, according to a representative embodiment.

[0022] Referring to FIG. 2, system 100 includes a workstation 105 for implementing and / or managing the processes described herein with regard to determining a procedure for retrieving a fragment of a fractured endovascular device from a subject (patient) 150 using medical images from a medical imaging device 140, such as an X-ray imaging device or an ultrasound imaging device, for example. The workstation 105 includes processor 120, memory 130, a user interface 122 and a display 124. The processor 120 communicates with the medical imaging device 140 through an imaging interface (not shown). The medical imaging device 140 may be an X-ray imaging device, such as a fixed or mobile C-arm X-ray device, with an X-ray tube adapted to generate X-rays and an X-ray detector configured to acquire X-ray images of the vasculature of the subject 150. The medical imaging device 140 may be an ultrasound imaging device, including an ultrasound transducer probe operable by a user to obtain ultrasound images of thevasculature of the subject 150. Of course, other types of interventional imaging devices may be incorporated without departing from the scope of the present teachings.

[0023] The memory 130 stores instructions executable by the processor 120. When executed, the instructions cause the processor 120 to implement one or more processes for retrieving a fragment of a fractured endovascular device from the subject 150 using medical images (e.g., interventional images) acquired by the medical imaging device 140. The medical images may be provided from the medical imaging device 140 in real-time or near real-time during an interventional procedure, or may be retrieved from storage (e.g., database 112). For purposes of illustration, the memory 130 is shown to include software modules, each of which includes sets of instructions, executable by the processor 120, corresponding to an associated capability of the system 100.

[0024] The processor 120 is representative of one or more processing devices, and may be implemented by a general purpose computer, a central processing unit (CPU), a digital signal processor (DSP), a graphical processing unit (GPU), a tensor processing unit (TPU), a computer processor, a microprocessor, a state machine, programmable logic device, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof, using any combination of hardware, software, firmware, hard-wired logic circuits, or combinations thereof. Any processor or processing unit herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices. The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems, such as in a cloud-based or other multi-site application. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

[0025] The memory 130 is representative of one or more memories, and may include main memory and / or static memory, where such memories may communicate with each other and the processor 120 via one or more buses. The memory 130 may be implemented by any number, type and combination of random access memory (RAM) and read-only memory (ROM), for example, and may store various types of information, such as software algorithms, artificialintelligence (Al) models, machine learning models, and computer programs, all of which are executable by the processor 120. The various types of ROM and RAM may include any number, type and combination of computer readable storage media, such as a disk drive, flash memory, an electrically programmable read-only memory (EPROM), an electrically erasable and programmable read only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD-ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, a universal serial bus (USB) drive, or any other form of storage medium. The memory 130 is a tangible storage medium for storing data and executable software instructions, and is non- transitory during the time software instructions are stored therein. As used herein, the term “non- transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The memory 130 may store software instructions and / or computer readable code that enable performance of various functions. The memory 130 may be secure and / or encrypted, or unsecure and / or unencrypted.

[0026] The system 100 may also include database 112 for storing information that may be used by the various software modules of the memory 130. For example, the database 112 may include image data from previously obtained medical images of the subject 150 and / or of other similarly situated subjects, having fragments of fractured endovascular devices in their vasculatures. The stored image data may be used for training a machine learning model (algorithm), such as a neural network, for example, as discussed below. The database 112 may be implemented by any number, type and combination of RAM and ROM, for example. The various types of ROM and RAM may include any number, type and combination of computer readable storage media, such as a disk drive, flash memory, EPROM, EEPROM, registers, a hard disk, a removable disk, tape, CD-ROM, DVD, floppy disk, Blu-ray disk, USB drive, or any other form of storage medium known in the art. The database 112 comprises tangible storage mediums for storing data and executable software instructions and is non-transitory during the time data and software instructions are stored therein. The database 112 may be secure and / or encrypted, or unsecure and / or unencrypted. For purposes of illustration, the database 112 is shown as a separate storage medium, although it is understood that it may be combined with and / or includedin the memory 130, without departing from the scope of the present teachings.

[0027] The processor 120 may include or have access to an artificial intelligence (Al) engine, which may be implemented as software that provides artificial intelligence (e.g., machine learning models) and applies machine learning described herein. The Al engine may reside in any of various components in addition to or other than the processor 120, such as the memory 130, an external server, and / or the cloud, for example. When the Al engine is implemented in a cloud (not shown), such as at a data center, for example, the Al engine may be connected to the processor 120 via the internet or other communication network using one or more wired and / or wireless connection(s).

[0028] The user interface 122 is configured to provide information and data output by the processor 120, the memory 130 and / or the medical imaging device 140 to the user and / or for receiving information and data input by the user. That is, the user interface 122 enables the user to enter data and to control or manipulate aspects of the processes described herein, and also enables the processor 120 to indicate the effects of the user’s input, which may include control or manipulation of the medical imaging device 140. All or a portion of the user interface 122 may be implemented by a graphical user interface (GUI), such as GUI 128 viewable on the display 124, discussed below. The user interface 122 may include one or more interface devices, such as a mouse, a keyboard, a trackball, a joystick, a microphone, a video camera, a touchpad, a touchscreen, voice or gesture recognition captured by a microphone or video camera, for example.

[0029] The display 124 may be a monitor such as a computer monitor, a television, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT) display, or an electronic whiteboard, for example. The display 124 includes a screen 126 for viewing medical images of the subject 150, along with various features described herein to communicate to the user recommendations of types of procedures to use for retrieving fragments of fractured endovascular device, together with predicted probabilities of successful outcome of retrieving the fragment using the recommended types of procedures, as well as levels of confidence (uncertainty) in predicted probabilities and / or recommendations of types of retrieval devices to be used when the type of procedures is an interventional procedure, together with predicted probabilities of successful outcome ofretrieving the fragment using the recommended types of retrieval devices. The screen 126 further enables viewing of the GUI 128 to enable the user to interact with the displayed images and features.

[0030] Referring to the memory 130, the various modules therein store sets of data and instructions executable by the processor 120 to determine a procedure for retrieving a fragment of a fractured endovascular device, as mentioned above. Medical imaging module 131 is configured to receive and process medical images of the subject 150. The medical images may be acquired non-invasively by the medical imaging device 140, and may be received from the medical imaging device 140 in real-time or near real-time during a contemporaneous imaging session of the subject 150. For example, the medical images may be intra-procedural angiography images acquired during an interventional procedure. Alternatively, or in addition, the medical images may be previously acquired images, obtained during previous imaging session(s), which have been retrieved from storage (e.g., database 112). The medical images show the fragments of the fractured endovascular device present in the vasculature of the subject 150, and may be displayed on the display 124.

[0031] Procedure prediction module 132 is configured to train and apply a machine learning model to input data, including extracted image features from the medical images, in order to recommend a type of procedure for retrieving a fragment of a fractured endovascular device from the vasculature of the subject 150 based on artificial intelligence. The machine learning model may be a deep learning model, and may be implemented as neural network, for example, such as an artificial neural network (ANN), a convolutional neural network (CNN), a recurrent neural network (RNN), or a U-net model, for example.

[0032] The machine learning model predicts, including based on the extracted image features, probability of successful outcome with regard to retrieving the fragment from the vasculature using multiple different types of procedures suitable for retrieving the fragment from the vasculature, respectively, and determines a preferred type of procedure from the multiple types of procedures as the one having the highest predicted probability of successful outcome with regard to retrieving the fragment from the vasculature. The machine learning model recommends the preferred type of procedure to the user, including “interventional procedure,” “surgical procedure,” and “no procedure.” “No procedure” is recommended when leaving thefragment in the patient has the highest probability of successful outcome. Alternatively, the machine learning model may recommend performance of no procedure, without predicting the probability of success of performing no procedure, when the predicted probabilities of success using any other procedure are unacceptably low. Alternatively, the machine learning model may make no recommendation when the predicted probabilities of success using any of the types of procedures are unacceptably low.

[0033] Ideally, “successful outcome” is a long-term outcome in which the patient has no (or manageable) adverse events and lifestyle limitations resulting from the fragment of the fractured endovascular device having been in the patient’s vasculature. Determining the probability of a successful outcome includes estimating physical injury to the patient by performing a procedure for complete retrieval of the fragment of the fractured endovascular device from the vasculature, estimating physical injury to the patient by leaving the fragment of the fractured endovascular device in the vasculature, estimating side effects (e.g., exposure to X-ray imaging, long hospital stays) of performing a procedure, and estimating long-term quality of life for the patient when the fragment is removed and when the fragment is left in the patient. Factors that may be considered for performing no procedure, in particular, may include likelihood of occurrence of adverse events, such as fragment migration through the vasculature, distal embolization caused by the fragment, and / or stroke caused by blockage by or migration of the fragment, for example. Quality of life is a known metric based on various factors, such as mobility, pain, medications, follow-up treatment, and the like. The predicted probability of successful outcome reflects the percentage chance of attaining this level of success.

[0034] In some embodiments, the machine learning model may also determine levels of confidence associated with its predictions of the probabilities of successful outcome of the types of procedures. The levels of confidence may factor into the determination of the preferred type of procedure, as well as the determination to recommend no procedure. In this regard, the predictions having higher levels of confidence are weighted more than the predictions having lower levels of confidence.

[0035] For ease of explanation, the procedure prediction module 132 is divided into training phase module 133 and inference (application) module 134. The training phase module 133 is configured to perform training of the machine learning model based on retrospective (historic)data from previous patients, and possibly the same patient during previous procedures, having fragments of fractured endovascular devices in their respective vasculatures. For instance, the training phase module 133 receives retrospective data from a relatively large patient population (e.g., greater than 100) of patients who have undergone endovascular thrombectomy procedures in which the endovascular devices fractured. The retrospective data includes annotated two dimensional (2D) medical images (e.g., angiographic images or image sequences) of the vasculature including fragments of fractured endovascular devices. Image features related to the fractured endovascular devices, such as the locations of the endovascular devices and / or the fragments may be annotated manually using any annotation tools, such as bounding box, for example, or automatically using any known image processing or computer vision approaches. Image features including vessel characteristics, such as diameter and tortuosity score, for example, may be manually or automatically extracted from the medical images.

[0036] The retrospective data further includes the types of retrieval devices respectively associated with the 2D images / sequences used for retrieving the fragments from the vasculatures. The data regarding the types of retrieval devices may also include device characteristics associated with the different types of retrieval devices, such as device diameter and flexibility, for example. The types of devices and (optionally) the retrieval device characteristics may be annotated manually, e.g., selected by user from a drop-down menu on the GUI 128, or automatically by looking up the associated procedure report or via image processing, e.g., by the processor 120. Automatic detection of the retrieval devices and device characteristics, as image features, may be done by performing object detection based on images captured by operating room cameras or the like before the retrieval devices are inserted into the patients, or alternatively based on medical images (e.g., 2D fluoroscopy images) after the retrieval devices are inserted into the patients, respectively.

[0037] The retrospective data further includes ground truth outcomes of the fragment removal efforts after retrieval device deployments in the patients. The ground truth outcomes may include patient outcome over different durations of time (e.g., 6-month, 2-year, etc.), including measures of adverse events and lifestyle limitations, if any, following the procedures (or no procedures). Outcome may also be demonstrated as binary outcomes, such as success or failure of actually retrieving the fragment, for the fragment removal efforts. Alternatively, or inaddition, the ground truth outcomes may be demonstrated as qualitative outcomes grading the level of success of the fragment removal efforts. For example, the qualitative outcomes may include removal results, such as “fragment successfully removed,” “fragment moved 2cm,” “failed to reach target vessel due to tortuosity,” or “fragment migrated.” Of course, other annotations indicating the levels of success and results of the previous retrieval efforts may be included without departing from the scope of the present teachings.

[0038] Optionally, the retrospective data may include information about the fractured endovascular device associated with each 2D image / sequence, as well as device characteristics, such as device diameter, material and flexibility, for example. Like the retrieval devices, the types of endovascular devices and characteristics may be annotated manually, e.g., selected by user from a drop-down menu on the GUI 128, or automatically by looking up the associated procedure report or via image processing, e.g., by the processor 120. Automatic detection of the endovascular devices and characteristics, as image features, may be done by performing object detection based on images captured by operating room cameras or the like, or alternatively based on medical images (e.g., 2D fluoroscopy images) after the endovascular devices are inserted into the patients, respectively. If available, the retrospective data may also optionally include information about the patients themselves that may affect efficacy and / or success rate of the retrieval processes. Such patient information may be retrieved from EHRs, and may include patient age, blood pressure, weight, heart health, smoking history, and family health history, for example.

[0039] The machine learning model is trained using the annotated retrospective data and ground truth data to predict the probabilities of successful outcome with regard to retrieving fragments of endovascular devices from the vasculatures using various types of procedures, to predict the probabilities of successful outcome with regard to retrieving fragments of endovascular devices from the vasculatures using various types of retrieval devices for interventional procedures, and to determine levels of confidence with regard to the predicted probabilities. For example, the machine learning model may determine and analyze relationships or correlations between the retrospective data to predict the probabilities of successful outcome with regard to retrieving fragments of the endovascular devices from the vasculatures using the various types of retrieval devices for interventional procedures and determine confidence levels.

[0040] With regard to training the machine learning model to predict the probabilities of successful outcome using types of procedures, the types of procedures may include one or more types of surgical procedures, one or more types of interventional procedures, and no procedure. Interventional procedures include endovascular procedures where devices are navigated within vasculatures, as well as endobronchial procedures where devices are navigated within airways. In an embodiment, the machine learning model may also predict the probabilities of further adverse events associated with the types of procedures, such as fragment migration, and perforations or vasospasms of the vasculature, for example. Another example of an adverse event is when a fragment is not completely detached from a device, but retrieval of the device may lead to detachment or harm to the patient, e.g., if the fragment is embedded into the vessel wall and may perforate the vessel. Generally, the type of procedure with the highest probability of successful outcome (and / or lowest probability of further adverse events) is provided by the machine learning model as the suggested approach. When the suggested type of procedure is an interventional procedure, the machine learning model may further provide a list of appropriate retrieval devices in order of highest to lowest probabilities of successful outcome.

[0041] When the machine learning model is implemented as a neural network, the neural network may include several fully connected or convolutional layers with pooling, normalization, and non-linear layers between them, for example. When optional information about the patients (e.g., from EHRs) is included, this information may be incorporated by appending it to flattened feature layers before applying linear transformations followed by nonlinearities, such as sigmoid functions and rectified linear units (ReLU), for example. The neural network may directly produce an nx 1 vector output, where n is the number of values regressed by the neural network, such as probabilities of the successful outcome of each type of procedure, including probabilities of adverse events and lifestyle limitations, and properties of suggested devices for next steps, for example. Errors are computed by comparing the output values produced by the neural network with ground truth values using some loss function, such as LI or L2 loss, Huber loss, or log cosh loss, for example. The loss functions may be used to perform stochastic gradient descent to optimize network weights. This is typically done by computing derivatives of the loss function with respect to the model parameters using the chain rule. The derivatives inform the network how the model parameters must be adjusted through each trainingloop in order to minimize the loss function. The adjustments to model parameters are made starting from the last layer and working backwards in the network towards the first layer, which is known as backpropagation.

[0042] With regard to determining levels of confidence (or uncertainties) in estimating the intervention types, the machine learning model may be trained to estimate the levels of confidence based on errors produced by the machine learning model’s estimate of the probabilities of successful outcome using various procedure types and retrieval devices and the corresponding ground truths. Alternatively, the machine learning model may be trained to estimate the levels of confidence based on errors produced by the entire estimated n l vectors. In addition, the neural network, in particular, may have dropout layers, which ignore the output of some number of nodes at random, producing slightly different results for the same input. The variance in the results generated from inputting the same input into the trained neural network multiple times may inform confidence. For example, higher variance implies lower confidence and lower variance implies higher confidence. The level of confidence may also be inferred from attention maps created by the neural network. When the attention of the neural network concentrates around the area where fragments are located, the neural network should have higher confidence, whereas when the attention of the neural network concentrates on the background or irrelevant tissues or vasculature, the neural network should have lower confidence.

[0043] The machine learning model may consist of a single neural network that estimates all outputs together. Alternatively, the machine learning model may consist of two cascaded neural networks, where a first neural network is used to suggest the procedure type and a second neural network is used to predict the type of retrieval device to be used when the first neural network suggests an interventional procedure for retrieval. The two networks may share weights and / or may be trained together via combined loss functions and a single backpropagation loop.

[0044] The inference module 134 is configured to apply the trained machine learning model to current input data from the subject 150. The input data includes image data of an medical image showing the fragment of fractured endovascular device in the subject’s vasculature. The image data may be from 2D angiographic images and sequences acquired by an X-ray imaging device or from 2D ultrasound images acquired by an ultrasound imagining device, for example, although other types of medical image data may be incorporated without departing from thescope of the present teachings. The inference module 134 may derive image features from the medical image that are used by the inference module 134 to evaluate characteristics related to the fragment and the types of procedures available for addressing the fragment, including using known processing techniques. The image features may include the number of fragments of the fractured endovascular device, the size, shape, orientation and location of each fragment, and characteristics of the vasculature surrounding each fragment (e.g., tortuosity, occlusion). The image features derived may depend on imaging modality and quality of the images. Optionally, the input data may further include device information that describes characteristics of the fractured endovascular device and / or patient information that describes characteristics of the subject 150. For example, the device information may include the diameter, material and flexibility of the fractured endovascular device. The patient information may include EHR and / or PHI data, such as patient age, blood pressure, weight, heart health, smoking history, and family health history, for example.

[0045] In an embodiment, the input data may further include identification of a region of interest (ROI) in the medical image. The ROI may be identified manually by the user via the user interface 122, or automatically by the processor 120 as a predetermined area surrounding the location of the fragment in the medical image. In this case, the machine learning model may be applied only to input data from the ROI. Alternatively, the machine learning model may be applied to all input data, but the input data is weighted based on proximity to the ROI. That is, the image data within the ROI has the greatest weight and image data outside the ROI is weighted less the further it is from the ROI. If such a weighted approach is used at inference, the training data must also be similarly weighted to generate a model that can process such weighted inputs.

[0046] Based on the input data, the trained machine learning model outputs predicted probabilities of successful outcome with regard to retrieving the fragment of the fractured endovascular device from the vasculature using multiple different types of procedures based on the input data. Generally, as discussed above, the predicted probability of successful outcome for each procedure is determined based on the likelihood of the fragment being removed, while not causing, or otherwise minimizing, additional adverse events and lifestyle limitations, such as damaging (e.g., perforating) the vasculature during the retrieval or causing limited mobility. Thepredicted probability of successful outcome may be determined by the trained model based on correlation of features and parameters related to the fragment in the input data and relationships between such features and parameters. As discussed above, the types of procedures include one or more types of interventional procedures, one or more types of surgical procedures, and no procedure. For example, interventional procedures may include endovascular procedures using associated types of retrieval devices. Based on the predicted probabilities of successful outcome, the machine learning model determines and outputs a preferred type of procedure from among the multiple available types of procedures. In particular, the machine learning model determines that the type of procedure producing the highest predicted probability of successful outcome with regard to retrieving the fragment from the vasculature is the preferred type of procedure.

[0047] In an embodiment, the machine learning model further determines and outputs a level of confidence associated with each of the predicted probabilities of successful outcome of the types of procedures. The levels of confidence may be based on how closely the input data matches sets of training data used to train the machine learning model for the different types of procedures, for example. In this case, when determining the preferred type of procedure, the machine learning model includes consideration of the levels of confidence associated with the predicted probabilities of successful outcome. For example, the types of procedures having predicted probabilities of successful outcome with higher confidence levels will be weighted heavier than the types of procedures having predicted probabilities of successful outcome with lower confidence levels. So, for example, when two or more types of procedures have the same or similar predicated probabilities of successful outcome, the one with the highest level of confidence will be determined as the preferred type of procedure. Also, predicted probabilities of successful outcome having confidence levels below a predetermined confidence threshold may simply be disregarded.

[0048] The machine learning model may recommend performing no procedure when performing no procedure has the highest predicted probability of successful outcome or when the highest predicted probability of successful outcome of the other types of procedures is less than a predetermined probability threshold. Likewise, the machine learning model may recommend performing no procedure when none of the predicted probabilities of successful outcome above the probability threshold have associated confidence levels over a predetermined confidencethreshold When none of the types of procedures, including performing no procedure, has a corresponding predicted probability of successful outcome and / or confidence level above corresponding probability or confidence thresholds, the machine learning model may provide no outcome and / or otherwise default to physician experience since the machine learning model is unable to generate a prediction with high enough confidence. For example, since none of the options have high probability of successful outcome or high confidence, including when no procedure in performed, the machine learning model may output a message indicating that it cannot make a recommendation.

[0049] As stated above, the different types of procedures include interventional (e.g., endovascular) procedures, surgical procedures, and no procedure. When the preferred type of procedure recommended by the machine learning model is an interventional procedure, the machine learning model may go on to predict probabilities of successful outcome of retrieving the fragment of the fractured endovascular device from the vasculature using multiple different types of retrieval devices based on the training data. The predicted probability of successful outcome for retrieval is determined based on the likelihood of the fragment being removed, while not causing, or otherwise minimizing, additional adverse events during the retrieval and after retrieval, such as damaging (e.g., perforating) the vasculature, and not causing, or otherwise minimizing, lifestyle limitations, as discussed above. The types of retrieval devices may include snares, aspiration or guide catheters, angioplasty balloons, or stent retrievers, for example.

[0050] Based on the predicted probabilities of successful outcome, the machine learning model determines and outputs a preferred type of retrieval device from among the multiple available types of retrieval devices. In particular, the machine learning model determines that the type of retrieval device producing the highest predicted probability of successful outcome with regard to retrieving the fragment from the vasculature using the preferred type of procedure is the preferred type of retrieval device. In an embodiment, the machine learning model may rank the types of retrieval devices from most to least likely to retrieve the fragment in a list, which may be displayed on the display screen 126 and / or the GUI 128. The user may then select the preferred type of retrieval device from the list, using user interface 122 and / or the GUI 128, which may or may not be the retrieval device with the highest predicted probability of successful outcome, depending on user preferences. For example, the user may have more experience witha particular type of retrieval device, which is different from the retrieval device with the highest predicted probability of successful outcome, and therefore select that particular type of retrieval device from displayed list.

[0051] In an embodiment, the machine learning model further determines and outputs a level of confidence associated with each of the predicted probabilities of successful outcome associated with the types of retrieval devices for the preferred type of procedure. In this case, when determining the preferred type of retrieval device, the machine learning model includes consideration of the levels of confidence associated with the predicted probabilities of successful outcome. For example, the types of retrieval devices having predicted probabilities of successful outcome with higher confidence levels will be weighted higher than the types of retrieval devices having predicted probabilities of successful outcome with lower confidence levels. So, for example, when two more types of retrieval devices have the same or similar predicated probabilities of successful outcome, the one with the highest level of confidence will be determined as the preferred type of retrieval devices. Also, predicted probabilities of successful outcome having confidence levels below a predetermined confidence threshold may simply be disregarded. In the event the types of retrieval devices are presented to the user in a list, as discussed above, the list may include the confidence levels together with predicted probabilities of successful outcome associated with retrieval devices. Again, the user may then make a more informed decision as to which retrieval device to select as the preferred type of retrieval device, which may or may not be the type of retrieval device with the highest predicted probability of successful outcome.

[0052] On the other hand, when the preferred type of procedure recommended by the machine learning model is a surgical procedure, the machine learning model may go on to determine a preferred incision site on the subject 150 for initiating the surgical procedure. The preferred incision site may be predicted based on probabilities of successful outcome of retrieving the fragment of the fractured endovascular device from the vasculature from multiple different incision sites on the subject 150 based on the training data. The predicted probability of successful outcome for retrieval is determined based on the likelihood of the fragment being removed, while not causing, or otherwise minimizing, str ess / discomfort to the subject 150 and additional adverse events, such as damaging the vasculature or surrounding tissue of the subject150, during the surgical retrieval. Training the machine learning model for identifying incision sites may be done by the training phase module 133 based on the surgical location identified in procedure reports, external images acquired during surgical procedures which show the incision sites, or medical images of the device fragment locations including landmarks such as skeletal structures, vascular structures, etc., which may be spatially associated with incision sites, for example.

[0053] Ground truth probability of successful outcome may be a difficult annotation to acquire since known information per patient only includes the procedure used and whether or not it resulted in a successful outcome. Probability of successful outcome may, however, be inferred by a model trained in an unsupervised setting. That is, in an alternative embodiment, the training phase module 133 may train the machine learning model for reconstructing the input data in order to learn a latent representation or embedding of input training data. This training is unsupervised because ground truth annotations are not required. Training for a reconstruction task may be performed using an encoder-decoder architecture, such as an auto-encoder or a variation auto-encoder, for example. The encoder is configured to compress the input data to a reduced dimensional latent embedding. The latent embedding is then used to reconstruct the input using the decoder. The machine learning model is trained to reconstruct the input from the latent embedding using a loss function to provide a reconstruction output, where the loss function compares the reconstructed output with the training data input using, for example, LI or L2 loss. The comparisons are then used to perform gradient descent and to update weights, as in the previous embodiment.

[0054] In this case, the inference module 134 uses the embedding space to determine the probabilities of successful outcome. FIG. 3 is a schematic diagram showing how embedding space of the trained machine learning model (e.g., neural network) is leveraged to make predictions at inference, according to a representative embodiment.

[0055] Referring to FIG. 3, the current input data goes through the encoder to generate a latent embedding z (square). By searching in the learned embedding space within a neighborhood of the latent embedding z, attributes of similar training samples may be retrieved. For instance, the example in FIG. 3 shows where endovascular procedures (triangles) and surgical procedures (circles) were used for fragment retrieval. In the depicted example, withinneighborhood 300 of the latent embedding z, 68 percent of the training data samples performed endovascular procedures for fragment retrieval, while 32 percent of the training data samples performed surgical procedures for fragment retrieval. Of the endovascular procedures, 73 percent were successful (triangles with dark outline) and the remaining 27 percent were unsuccessful. The value of 73 percent may be outputted directly by the machine learning model as the probability of successful outcome using an endovascular procedure or the outputted probability may be weighted by the proximity to the latent embedding z. That is, if successful samples were further away from the latent embedding z in embedding space, then the probability of successful outcome for the latent embedding z should be lower, even if 73 percent of all samples within the neighborhood 300 successfully retrieved the fragment using an endovascular procedure. This is because a larger distance in embedding space denotes some difference between the latent embedding z and the training data samples, which may originate from differences in imaging features or patient characteristics, for example. Alternatively, this distance may also be used to compute a confidence level in the success prediction.

[0056] Similarly, of the successful endovascular procedures, 64 percent used a snare as the retrieval device, and 36 percent used an angioplasty balloon as the retrieval device. The percentage risks associated with each are also shown (e.g., migration, perforation, vasospasm). Similar calculations may be made for the surgical procedure or other types of endovascular procedures available in the training data. The alternative embodiment allows the machine learning model to learn an embedding space by training to perform a reconstruction of input data, and leveraging known attributes of training data to compute probabilities and inform decisions for new inputs at inference.

[0057] Referring again to FIG. 2, visualization module 135 is configured to interface with the display 124 for visualizing information for the user, including one or more outputs from the machine learning model of the intervention prediction module 132, and optionally one or more medical images from medical imaging module 131. The visualized information displayed on the display 124 may include the preferred type of procedure and, when the preferred type of procedure is an interventional procedure, the preferred type of retrieval device, together with the respective estimated confidence levels, respectively. As mentioned above, the types of interventional devices may be displayed in a list, rated from the highest to lowest probabilities ofsuccessful outcome for the interventional retrieval. In an embodiment, the list may be checked against a lookup table containing available retrieval devices at the facility, so that the visualization module 135 displays only those retrieval devices that are actually available at the site. When the preferred type of procedure is a surgical procedure, the visualization module 135 may cause the display 124 to show the preferred incision site on the subject 150 as determined by the machine learning model, discussed above. For example, the preferred incision site may be depicted in a displayed 2D or 3D animation or model of the subject 150 for clarity.

[0058] FIG. 4A and FIG. 4B provide a flow diagram of a method for determining a workflow with regard to a fragment of a fractured endovascular device in a vasculature of a patient, according to a representative embodiment. FIGs. 4A and 4B may be implemented by the processor 120 of the workstation 105, executing instructions stored in the memory 130, for example. Further, the steps of FIGs. 4A and 4B are described in the context of retrieving one fragment of a fractured endovascular device from the vasculature of the patient for the sake of convenience, although it is understood that they apply equally to multiple fragments at the same or multiple locations.

[0059] Referring to FIG. 4A, the method includes training a machine learning model in block S411 using retrospective training data, as discussed above. The training may include receiving the retrospective training data from a database (e.g., database 116) collected from previous fragment retrieval procedures. The retrospective data includes input training data and ground truth data corresponding to the input training data. The input training data includes medical images of vasculatures from a training set of patients, where the medical images or sequences of medical images show fragments of fractured endovascular devices. The medical images may be annotated. The input training data further includes information regarding the types of retrieval devices used for retrieving the fragments from the vasculatures shown in the medical images, whenever the retrieval processes were performed as interventional procedures (as opposed to surgical procedures). The input training data may further include information about the fractured endovascular devices associated with the images, including device characteristics, and information about the patients themselves that may affect efficacy and / or success rate of the retrieval processes, including patient information may be retrieved from EHRs, for example.

[0060] The retrospective data further includes ground truth outcomes of the fragment 1removal efforts of the training set of patients after the retrieval device deployments. The ground truth outcomes may be qualitative outcomes describing the end results of the fragment removal efforts, respectively, associated with the corresponding input training data. The machine learning model is trained using the input training data and the corresponding ground truth outcomes to determine predictions of successful outcomes with regard to retrieving the fragments of the fractured endovascular devices associated with each of the types of procedures, including no procedure, respectively, and to identify preferred types of retrieval devices when the types of procedures are interventional procedures, as discussed above.

[0061] In an embodiment, the machine learning model is further trained to estimate levels of confidence of successfully retrieving the fragments of the fractured endovascular devices using the recommended types of procedures and the types of retrieval devices (when appropriate). The training may be based on errors produced by the machine learning model’s estimates of the probabilities of successful outcome using various intervention types and retrieval devices as compared to the corresponding ground truths. Alternatively, the training may be based on errors produced by the entire estimated n l vectors. The neural network, in particular, may have dropout layers, which ignore the output of some number of nodes at random, producing slightly different results for the same input. The variance in the results generated from inputting the same input into the trained neural network multiple times may inform confidence.

[0062] In block S412, receiving input data with regard to the fragment of the fractured endovascular device in the vasculature of the patient. The input data includes a medical image of the vasculature showing the fragment of the endovascular device. The medical image may be an angiographic X-ray image of the vasculature, for example, although other types of medical images capable of showing the fragment of the fractured endovascular device, such as ultrasound images and CT scans, may be utilized without departing from the scope of the present teachings. The medical image includes image features that may be used for evaluating the fragment and the types of procedures, where the image features may depict the size, shape, orientation and location of the fragment of the fractured endovascular device, and characteristics of the vasculature surrounding the fragment. In various embodiments, the input data may further include device information describing the fractured endovascular device and / or patient information providing patient age, blood pressure, weight, heart health, smoking history, familyhealth history, and other relevant information, e.g., retrieved from EHRs.

[0063] If a biplane C-arm system is being used as the medical imaging device during the retrieval procedure, then the machine learning model may use inputs from the two different views to make predictions during training and inference. For example, the medical images from the two views may be input as two input channels in a single machine learning model.Alternatively, the medical images may be provided as separate inputs into two identical machine learning models set up in a Siamese network architecture, where the two identical machine learning models converge at a later layer, such that they share the same weights and parameters in the later layers (i.e., both networks inform each other). In another embodiment, the machine learning model may receive 3D images of the vasculature, when available, as an additional input medical image, which improves prediction based on the additional information about vascular geometries provided by the 3D images.

[0064] In block S413, the trained machine learning model is applied to the input data received in block S412, including at least the medical image of the vasculature showing the fragment. In response, the machine learning model performs the steps described below with reference to blocks S414 to block S425, discussed below.

[0065] In block S414, predictions of probabilities of successful outcome with regard to retrieving the fragment from the vasculature using multiple types of procedures, respectively, are determined. The types of procedures include one or more interventional procedures, one or more surgical procedures, and no procedure. The predictions are based on the training data, discussed above.

[0066] In block S415, a level of confidence associated with each of the multiple types of procedures is optionally determined. The level of confidence reflects the accuracy of the predictions of the probabilities of successful outcome, which may be based on factors such as how closely the input data received in block S412 matches sets of training data used to train the machine learning model for the different types of procedures, for example.

[0067] In block S416, a preferred type of procedure is determined from the multiple types of procedures. The preferred type of procedure may be the one having the highest predicted probability of successful outcome with regard to retrieving the fragment from the vasculature. The preferred type of procedure may be an interventional procedure, a surgical procedure, or noprocedure (when leaving the fragment in the patient vasculature has the highest probability of successful outcome). When the levels of confidence associated with the types of procedures are also determined, determining the preferred type of procedure is further based on consideration the levels of confidence associated with the types of procedures. For example, types of procedures having higher levels of confidence may be weighted more heavily than types of procedures having lower levels of confidence. Accordingly, the type of procedure having the highest predicted probability of successful outcome may not be identified as the preferred type of procedure if it has an unacceptably low confidence level or another type of procedures has a slightly lower probability of successful outcome but a higher confidence level, for example.

[0068] In an embodiment, medical images and / or image sequences from a current attempt to retrieve the fragment may be inputted into the machine learning model to continually update the risk prediction during the retrieval. In this case, if all potential procedures show unacceptably high risk at any point during the retrieval attempt, the retrieval attempt may be abandoned before causing the subject additional harm, as discussed below.

[0069] After determining the preferred type of procedure in block S416, the process may continue to block S417 in FIG. 4B (indicated by “A”), where it is determined whether the preferred type of procedure determined in block S416 is acceptable from a risk standpoint for a successful outcome of the procedure with regard to retrieving the fragment of the fractured endovascular device. The determination may be made by comparing the predicted probability of successful outcome with a predetermined probability threshold, for example, and determining that the preferred type of procedure is not acceptable when it is below the predetermined probability threshold. Alternatively, or in addition, the determination may include comparing the level of confidence associated with the predicted probability of successful outcome with a predetermined confidence threshold, for example, and determining that the preferred type of procedure is not acceptable when the level of confidence is below the predetermined confidence threshold.

[0070] When it is determined that the preferred type of procedure is not acceptable (block S417: No), the process proceeds block S418, outputting “no recommendation” with regard to an appropriate workflow to perform. For example, the probability of successful outcome may consider the likelihood of occurrence of adverse events, such as fragment migration through thevasculature, distal embolization caused by the fragment, and / or stroke caused by blockage by or migration of the fragment, and determine that the probability of successful outcome associated with any of the types of procedures is below a predetermined probability threshold. The “no recommendation” output may then be displayed in block S426, notifying the physician that the machine learning model is unable to generate a prediction with high enough probability of successful outcome and / or confidence, thereby defaulting to the physician’s experience.

[0071] When the preferred type of interventional procedure is acceptable (block S417: Yes), the process proceeds block S419, where it is determined whether the preferred type of procedure is an interventional procedure (e.g., endovascular procedure), a surgical procedure, or no procedure. When the preferred type of procedure is a surgical procedure (block S419: S), the process proceeds to block S420, recommending the surgical procedure as the appropriate workflow with regard to retrieving the fragment. The process then proceeds to block S426 in order to display the recommendation of “surgical procedure” as the output result. In an embodiment, the machine learning model may also determine a preferred incision site on the subject for initiating the surgical procedure (not shown). The preferred incision site may be predicted based on probabilities of successful outcome of retrieving the fragment of the fractured endovascular device from the vasculature from multiple different incision sites based on the training data.

[0072] When the preferred type of procedure is no procedure (block S419: N), the process proceeds to block S421, recommending that neither a surgical procedure nor an interventional procedure be performed in an attempt to retrieve the fragment. In this case, it is determined that the probability of successful outcome is acceptable if the fragment is left in the vasculature (i.e., no procedure is performed). When the probability of successful outcome associated with not performing a procedure is acceptable, the process proceeds to block S421, recommending “no procedure” as the appropriate workflow with regard to retrieving the fragment. The process then proceeds to block S426 in order to display the recommendation of “no procedure” as the output result.

[0073] When the preferred type of procedure is an interventional procedure (block S419: 1), the process proceeds to block S422 where probabilities of successful outcome of retrieving the fragment are determined using multiple different types of retrieval devices for performing thedetermined preferred type of interventional procedure based on the training data. In block S423, levels of confidence associated with the probabilities of successful outcome predicted for the types of retrieval devices may be optionally determined as well, also based on the training data.

[0074] In block S424, the preferred type of retrieval device for performing the interventional procedure is recommended. The preferred type of retrieval device is the one having the highest predicted probability of successful outcome for retrieving the fragment from the vasculature during the interventional procedure. When the levels of confidence associated with the types of retrieval devices are also determined, the recommended preferred type of retrieval device is further based on consideration the levels of confidence associated with the types of retrieval devices. For example, types of retrieval devices having higher levels of confidence may be weighted more heavily than types of retrieval devices having lower levels of confidence. In block S425, the interventional procedure is output as the recommended type of procedure and the preferred type of retrieval device is output as the recommended type of retrieval device for performing the interventional procedure. The process then proceeds to block S426 in order to display the recommendation of “interventional procedure” as the output result, along with the preferred type of retrieval device for performing the interventional procedure. In an embodiment, the types of retrieval devices may be displayed in a list in order of highest to lowest predicted probabilities of successful outcome in performing the preferred type of interventional procedure, where the preferred type for retrieval device is first on the list. In an embodiment, in the event of an unsuccessful retrieval effort using the highest rated retrieval device in the list, the suggested retrieval device list may be updated according to the new information indicating the failed attempt.

[0075] As indicated above, output results are displayed in block S426. In basic form, the output results identify the preferred type of procedure, and optionally an incision site for a surgical procedure and a preferred type of retrieval device for an interventional procedure. The user is then able to implement the preferred type of procedure as part of the medical treatment of the subject. The output results may further include information used to determine the preferred type of procedure, such as the corresponding predicted probability of successful outcome of the preferred type of procedure, as well as the level of confidence in the probability of successful outcome. In alternative embodiments, additional information from the machine learning modelmay also displayed for informational purposes, such as a list of all potential procedures along with the predicted probabilities of successful outcome and the corresponding levels of confidence. The user may then decide to implement the preferred or available type of procedure, or select an alternative workflow, such as an alternative procedure, based on the additional information.

[0076] Although the present specification describes components and functions that may be implemented in particular embodiments with reference to particular standards and protocols, the disclosure is not limited to such standards and protocols. Such standards are periodically superseded by more efficient equivalents having essentially the same functions. Accordingly, replacement standards and protocols having the same or similar functions are considered equivalents thereof.

[0077] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

[0078] One or more embodiments of the disclosure may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept. Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.

[0079] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. A system for determining a workflow with regard to a fragment of a fractured endovascular device in a vasculature of a patient, the system comprising: a processor in communication with memory, the processor configured to: receive at least one medical image of the vasculature, including the fragment of the fractured endovascular device; extract at least one image feature from the at least one medical image; obtain a plurality of types of procedures for retrieval of an object from the vasculature; and determine a type of procedure, from the plurality of types of procedures, to retrieve the fragment from the vasculature based on the at least one image feature.

2. The system of claim 1, wherein the processor is further configured to apply a machine learning model configured to: receive input data, including the at least one image feature; predict, for each of the plurality of types of procedures, a probability of a successful outcome of retrieval of the fragment from the vasculature based on the input data; and determine the type of procedure to retrieve the fragment from the vasculature based on the predicted probability for each of the plurality of types of procedures.

3. The system of claim 2, wherein the machine learning model is further configured to: determine a level of confidence associated with the predicted probability of the successful outcome for each of the plurality of types of procedure.

4. The system of claim 2, wherein the machine learning model comprises a neural network.

5. The system of claim 1, wherein the plurality of types of procedures comprises an interventional procedure, a surgical procedure, and no procedure.

6. The system of claim 5, wherein the processor is further configured to: if the determined type of procedure is the interventional procedure: obtain a plurality of types of retrieval devices for retrieval of the object from the vasculature; and determine a type of retrieval device, from the plurality of types of retrieval devices, to perform the determined type of procedure to retrieve the fragment from the vasculature based on the at least one image feature.

7. The system of claim 6, wherein the processor is further configured to: determine an ordered preference of the plurality of types of retrieval devices for retrieval of the fragment from the vasculature; display a list of the plurality of types of retrieval devices in the ordered preference; and enable a user to select one of the displayed retrieval devices from the displayed list.

8. The system of claim 6, wherein the processor is further configured to: if the determined type of procedure is the surgical procedure, determine a preferred incision site on the patient for initiating the surgical procedure to retrieve the fragment from the vasculature based on the at least one image feature.

9. The system of claim 2, further comprising a second processor configured to train of the machine learning model, the second processor configured to: receive training data comprising (i) previous medical images of vasculatures of previous patients including fragments of fractured endovascular devices in the previous medical images, (ii) types of procedures used for retrieving the fragments of the fractured endovascular devices corresponding to the previous medical images, and (iii) types of retrieval devices used for retrieving the fragments of the fractured endovascular devices corresponding to the previous medical images; training the machine learning model using the training data to predict probabilities of successful outcomes of retrieving the fragments of the endovascular devices associated with each of the types of procedures using each of the types of retrieval devices.

10. The system of claim 1, wherein the at least one medical image comprises an angiographic X-ray image.

11. The system of claim 1, wherein the processor is further configured to: receive device information describing the fractured endovascular device; and determine the type of procedure, from the plurality of types of procedures, based on the device information.

12. The system of claim 11, wherein the processor is further configured to: receive patient information providing at least one of age, clinical parameters, smoking history, and family history of the patient; and determine the type of procedure, from the plurality of types of procedures, based on the patient information.

13. The system of claim 1, wherein the processor is further configured to: identify a region of interest (ROI) in the at least one medical image of the vasculature; extract the at least one image feature only from the ROI or weight contribution of the at least one image feature in determination of the type of procedure based on proximity to the ROI.

14. A computer-implemented method of determining a workflow with regard to a fragment of fractured endovascular device in a vasculature of a patient, the method comprising: receiving at least one medical image of the vasculature, including the fragment of the fractured endovascular device; extracting at least one image feature from the at least one medical image; obtaining a plurality of types of procedures for retrieval of an object from the vasculature; and determining a type of procedure, from the plurality of types of procedures, to use to retrieve the fragment from the vasculature based on the at least one image feature.

15. The processor implemented method of claim 14, wherein determining the type of procedure includes applying a machine learning model comprising: receiving input data, including the at least one image feature; predicting, for each of the plurality of types of procedures, a probability of a successful outcome of retrieval of the fragment from the vasculature based on the input data; and determining the type of procedure to retrieve the fragment from the vasculature based on the predicted probability for each of the plurality of types of procedures.

16. The method of claim 15, wherein applying the machine learning model further comprises: determining a level of confidence associated with the predicted probability of successful outcome for each of the plurality of types of procedure.

17. The method of claim 14, wherein the plurality of types of procedures comprises at least one interventional procedure, a surgical procedure, and no procedure.

18. The method of claim 17, wherein: if the determined type of procedure is the interventional procedure, obtaining a plurality of types of retrieval devices suitable to retrieve the object from the vasculature; and determining a type of retrieval device, from the plurality of types of retrieval devices, to perform the determined type of procedure to retrieve the fragment from the vasculature based on the at least one image feature.

19. The method of claim 17, wherein: if the determined type of procedure is the surgical procedure, determining a preferred incision site on the patient for initiating the surgical procedure to retrieve the fragment from the vasculature based on the at least one image feature.

20. A non-transitory computer readable medium having stored instructions for determining a workflow with regard to a fragment of fractured endovascular device in a vasculature of a patient, the instructions, when executed by a processor, cause the processor to: receive at least one medical image of the vasculature, the at least one medical image including the fragment of the endovascular device; extract at least one image feature from the at least one medical image; and obtain a plurality of types of procedures for retrieval of an object from the vasculature; and determine a type of procedure, from the plurality of types of procedures, to retrieve the fragment from the vasculature based on the at least one image feature.

Images