System and method for estimating contrast agent usage

Abstract

A method for estimating contrast agent usage by a user during an interventional procedure includes receiving image data from a series of images acquired by a medical imaging device (130) during the interventional procedure (S311), where the series of images includes a portion of vasculature of the patient (150) through which the contrast agent flows, estimating the amount of the contrast agent based on the received image data (S313), and reporting the estimated amount of contrast agent to the user (S315). The amount of the contrast agent is estimated by applying the image data to a machine learning model (122) that has been trained based on features of previous image data associated with known amounts of contrast agents (S312), and estimating the amount of the contrast agent used based on similarity of features of the received image data and the features of the previous image data (S313).

Description

SYSTEM AND METHOD FOR ESTIMATING CONTRAST AGENT USAGEBACKGROUND

[0001] In various interventional procedures, contrast agent is injected using a manual injector (e.g., syringe) into the patients in order to enhance visibility provided by medical imaging, which typically provides guidance during the intervention procedure. It is not possible to accurately identify the contrast usage from a manual injector during such procedures. To this end, automated contrast agent injectors may be used to inject the contrast agent, in which case the amount of contrast agent is automatically tracked. However, it is still common to use manual contrast agent injectors, which are manually operated by the interventionist on an as needed basis.

[0002] In particular, the manual contrast agent injectors do not track the amount and / or timing of the contrast agent being injected into the patient. For example, it is not clear how much contrast agent is mixed with saline, how much contrast agent remains in the injector unadministrated, and how much contrast agent has leaked or drained from valves and connectors. Consequently, the interventionists can only approximate contrast usage during the interventional procedure and in subsequent procedure reports that they prepare, and are not able to otherwise track their contrast agent utility over time.

[0003] Without a systematic way of tracking these utilities, the interventionalists cannot easily track their performance of these utilities over time and / or identified key areas of improvement, such as procedure performance, efficiency, clinical outcomes, and the like.SUMMARY

[0004] According to a representative embodiment, a method is provided for estimating contrast agent usage by a user during an interventional procedure on a patient. The method includes receiving image data from a series of images acquired by a medical imaging device during the interventional procedure, where the series of images show a portion of vasculature of the patient through which contrast agent flows; applying the received image data to a first machine learning model that has been previously trained based on features of previous imagedata associated with known amounts of contrast agents; estimating, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data; and reporting the estimated total amount of the contrast agent to the user. In some embodiments, the reporting is via a display.

[0005] According to another representative embodiment, a system is provided for estimating contrast agent usage by a user during an interventional procedure on a patient. The system includes a medical imaging device configured to acquire a series of images during the interventional procedure, where the series of images show a portion of vasculature of the patient through which contrast agent flows; a control unit including at least one processor and a non- transitory memory storing instructions that, when executed, cause the at least one processor to: receive image data from the series of images acquired by the medical imaging device during the interventional procedure; apply the received image data to a first machine learning model that has been previously trained based on features of previous image data associated with known amounts of contrast agents; estimate, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data; and report the estimated amount of contrast agent to the user. In some embodiments, a display is coupled to the controller and configured to display the reporting.

[0006] According to another representative embodiment, a computer readable medium stores instructions for estimating contrast agent usage by a user during an interventional procedure on a patient. When executed, the instructions cause at least one processor to receive image data from a series of images acquired by a medical imaging device during the interventional procedure, where the series of images show a portion of vasculature of the patient through which contrast agent flows; apply the received image data to a first machine learning model that has been previously trained based on features of previous image data associated with known amounts of contrast agents; estimate, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data; and cause the estimated amount ofcontrast agent to be reported to the user. In some embodiments, the instructions cause at least one processor to provide the reporting via a display.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.

[0008] FIG. 1 is a simplified block diagram of a system for estimating contrast agent usage by a user during an interventional procedure on a patient, according to a representative embodiment.

[0009] FIG. 2 is an example of a display showing analysis of contrast agent usage by a user, according to a representative embodiment.

[0010] FIG. 3 is a flow diagram of a method of estimating contrast agent usage by a user during an interventional procedure on a patient, the method comprising, according to a representative embodiment.DETAILED DESCRIPTION

[0011] In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with the representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only, and is not intended to be limiting. The defined terms are in addition to the technical and scientific meanings of the defined terms as commonly understood and accepted in the technical field of the present teachings.

[0012] It will be understood that, although the terms first, second, third etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0013] The terminology used herein is for purposes of describing particular embodiments only, and is not intended to be limiting. As used in the specification and appended claims, the singular forms of terms “a,” “an” and “the” are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms “comprises,” and / or “comprising,” and / or similar terms when used in this specification, specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0014] Unless otherwise noted, when an element or component is said to be “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0015] In view of the foregoing, the present disclosure, through one or more of its various aspects, embodiments and / or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below. For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. However, other embodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Moreover, descriptions of well-knownapparatuses and methods may be omitted so as to not obscure the description of the example embodiments. Such methods and apparatuses are within the scope of the present disclosure.

[0016] Generally, various embodiments are directed to the technical field of medical imaging, and provide improvements in tracking and characterizing amounts of contrast agent injected into patients during interventional procedures that require medical imaging. An artificial intelligence (Al)-based control unit applies one or more machine learning models to longitudinal, time-series imaging data from interventional procedures from multiple sources (multi-modal data) by the same user and generates user-specific insights. The machine learning model(s) may be neural network model(s) that are trained using medical images and contrast injector data (e.g., injector size and speed) to estimate amounts of contrast agent injected manually per procedure for the user. The machine learning model(s) output results that may be displayed, e.g., in a dashboard, as contrast agent utility for the user over multiple procedures. The contrast agent utility may be categorized based on various aspects / tags, such as type of procedure, complications, interventional team, and duration of the interventional procedure, for example. The embodiments may be implemented as a software feature in medical imaging systems, such as fixed or mobile C-arm systems, for example, and / or as a standalone software feature that communicates with systems or data stores containing interventional imaging data, contrast injector data, and other information pertaining contrast agent usage.

[0017] FIG. 1 is a simplified block diagram of a system for estimating contrast agent usage by a user during an interventional procedure of a patient, according to a representative embodiment.

[0018] Referring to FIGI, system 100 includes a control unit 110, a medical imaging device 130, and a manual contrast agent injector 140, all of which are used for performing an interventional procedure on patient 150. The control unit 110 is configured to implement and / or manage the processes described herein during the interventional procedure performed by medical personnel, including representative user 152 (e.g., physician, interventionalist, technician, or other clinician operating all or part of the control unit 110) on the patient 150. The control unit 110 includes one or more processors indicated by processor 112, a user interface (IF) 114, and a display 116, and one or more memories indicated by memory 120. The medical procedure involves live imaging of the patient 150 by the medical imaging device 130. The interventionalprocedure may be any type of medical procedure that includes use of real-time medical imaging, e.g., by the medical imaging device 130, along with injectable contrast agent to assist in visualizing the vasculature and guiding interventional devices (e.g., surgical instruments, catheters, biopsy tools, implants, ultrasound transducers, and the like). Such interventional procedures may include an interventional endovascular procedures (e.g., heart catheterization and transcatheter aortic valve replacement (TAVR), mechanical thrombectomy, intra-cranial aneurysm treatment, etc.) and endobronchial procedures, for example. The interventional procedures may be performed all or in part using a surgical / interventional robot 155, although not necessarily.

[0019] The medical imaging device 130 may be any type of imaging device configured to perform real-time medical imaging using contrast agent without departing from the scope of the present teachings. For example, the medical imaging device 130 may be an x-ray imaging device for performing digital subtraction angiography (DSA), computed tomography angiography (CTA), or angiography, for example. Other types of medical imaging device 130 may be a magnetic resonance imaging (MRI) device and a positron emission tomography (PET) scan device, for example, without departing from the scope of the present teachings. All or part of the operations of the medical imaging device 130 are performed under control of the control unit 110 via a known imaging interface 135.

[0020] As an example, an x-ray imaging device, in particular, may have a rotatable and translatable C-arm, and an x-ray source and an x-ray detector connected in a fixed relationship at opposite ends of the C-arm. The x-ray source emits ionizing radiation, according to settings, such as dosage, frame rate, exposure time, and beam collimation, for example, that travels through a region of interest (ROI) 156 on the patient 150. The ROI 156 may be part of the patient’s anatomy, an operation site, or an access site for the interventional procedure, for example. The x- ray detector receives the x-ray radiation and acquires x-ray images in response for enabling visualization of the ROI 156, e.g., shown on the display 116. The images may include DSA images, fluoroscopy sequences, 3D roadmaps, and cone beam computed tomography (CBCT) images, for example. The C-arm is maneuverable for changing the location of the x-ray source relative to the patient 150 to accommodate a variety of different viewing angles for acquiring the images. The C-arm system can be fixed, mobile, motorized, or non-motorized.

[0021] The contrast agent injector 140 includes a manual injector, such as a syringe, for example, which contains a predetermined amount of contrast agent. Alternatively, the contrast agent injector may be an automatic injector, which automatically tracks the contrast agent usage. In this case, the contrast agent usage estimated by the control unit 110 may be used to enhance measurements used for the automatic tracking. The type of contrast agent depends on the medical imaging technology. For example, radiopaque iodine may be used for x-ray and CT imaging, gadolinium-based contrast agent (GBCA) may be used for MRI imaging, and Fluorodeoxyglucose (FDG) may be used for PET scanning. The contrast agent injector 140 may be inserted directly into the vasculature of the patient 150, e.g., via a needle attached to the syringe, or may be connected via a tube or catheter to a port that has already been placed in the patient 150. A plunger of the syringe, for example, may be slowly depressed by the user 152 in order to inject the contrast agent intravenously into the vasculature, resulting in a subjective amount of contrast agent entering the vasculature each operation of the contrast agent injector 140. Alternatively, control of the contrast agent injector 140 may be partially manual when it is connected to a port in the patient 150. In this case, the contrast agent injector 140 may include a pump or other automated device that is controlled via a known injector interface 145, e.g., such as a control knob, in response to manual controls entered by the user 152. The manual contrast agent injector 140 does not have a set flow rate or volume of contrast agent, so the flow rate and volume vary widely among different users and / or among various procedures performed by the same or different users. The time and exact amount of contrast agent is tracked by the automated contrast agent injector.

[0022] The memory 120 of the control unit 110 stores instructions executable by the processor 112. When executed, the instructions cause the processor 112 to implement one or more processes for estimating an amount of contrast agent injected by the user 152 into the patient 150 during the interventional procedure.

[0023] The processor 112 is representative of one or more processing devices, and may be implemented by a general purpose computer, a central processing unit (CPU), a computer processor, digital signal processor (DSP), a graphics processing unit (GPU), a microprocessor, a microcontroller, a state machine, programmable logic device, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof, using anycombination of hardware, software, firmware, hard-wired logic circuits, or combinations thereof. Any processing device or processor herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices. The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. A processor may also refer to a collection of processors within a single computer system or distributed among multiple computer systems, such as in a cloud-based or other multi-site application. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

[0024] The memory 120 may include main memory and / or static memory, where such memories may communicate with each other and the processor 112 via one or more buses. For purposes of illustration, the memory 120 is shown to include software modules, each of which includes the instructions corresponding to an associated capability of the control unit 110, as discussed below. The memory 120 may be implemented by any number, type and combination of random access memory (RAM) and read-only memory (ROM), for example, and may store various types of information, such as software algorithms, artificial intelligence (Al) machine learning models, and computer programs, all of which are executable by the processor 112. The various types of ROM and RAM may include any number, type and combination of computer readable storage media, such as a disk drive, flash memory, an electrically programmable readonly memory (EPROM), an electrically erasable and programmable read only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD- ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, a universal serial bus (USB) drive, a solid state drive (SSD), or any other form of storage medium known in the art. The memory 120 is a tangible storage medium for storing data and executable software instructions, and is non-transitory during the time software instructions are stored therein. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The memory 120 may store software instructions and / or computer readable code that enable performance of various functions. Thememory 120 may be secure and / or encrypted, or unsecure and / or unencrypted.

[0025] The processor 112 and the memory 120 may include or have access to one or more Al engines or modules, which may be implemented as software that provides artificial intelligence and machine learning algorithms, such as neural network modeling, described herein. The one or more Al engines may reside in any of various components in addition to or other than the processor 112, such as the memory 120, an external server, and / or the cloud, for example. When the one or more Al engines are implemented in a cloud, such as at a data center, for example, they may be connected to the processor 112 via the internet using one or more wired and / or wireless connection(s).

[0026] Database 128 stores additional data, which may be used for training the various machine learning algorithms discussed herein. For example, the database 128 may store historical data that is relevant to previous interventional procedures performed by the same or different users. The historical data includes information regarding types of interventional procedures, identification of users performing the interventional procedures, identification of medical imaging devices used for acquiring medical images during the interventional procedures, image data from the medical images acquired during the interventional procedures, and patient data. The historical data may be specific to particular patients, procedures, interventionalists and / or devices, or may be unspecific.

[0027] The user interface 114 is configured to provide information and data output by the processor 112 and / or the memory 120 to the user 152 and / or to provide information and data input by the user 152 to the processor 112 and / or the memory 120. That is, the user interface 114 enables the user 152 to enter data and to control or manipulate aspects of the processes described herein, and to control or manipulate aspects of the medical imaging and contrast agent injection. The user interface 114 also enables the processor 112 to indicate the effects of the user’s control to the user 152.

[0028] All or a portion of the user interface 114 may be implemented by a graphical user interface (GUI), such as GUI 119 on a touch screen 118 of the display 116, for example. The user interface 114 includes push buttons operable (pushed) by the user to initiate various commands for manipulating the displayed image, making measurements and calculations, and the like during an imaging session (e.g., CBCT or other x-ray examination) or at any pointduring the interventional procedure. The push buttons may be displayed by the GUI 119 on the touch screen 118, or may be physical buttons, for example. The user interface 114 may further include any other compatible interface devices, such as a mouse, a keyboard, a trackball, a joystick, microphone, a video camera, a touchpad, or voice or gesture recognition captured by a microphone or video camera, for example.

[0029] The display 116 may be any monitor capable of displaying the medical images (e.g., fluoroscopy images, x-ray images) and other information, including estimated contrast agent amounts and associated usage data, as discussed below. Examples of such a display 116 include a computer monitor, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, or a solid-state display. The display 116 may include the touch screen 118 and the GUI 119 to enable the user to interact with the displayed images and other features, as discussed herein.

[0030] As mentioned above, the memory 120 is depicted as including a number of memory modules for purposes of explanation. Included in the memory 120 is image data module 121. The image data module 121 is configured to extract image data from a series of images (e.g., fluoroscopy images) of the ROI 156 acquired by the medical imaging device 130 during the interventional procedure. The image data includes pixel information providing size, brightness, and color of pixels in the images. The pixel information may be used for identifying landmarks in the images, such as edges, corners, and bifurcations, for example, using known edge detection, segmentation, tracking, and feature detection techniques. The image data module 121 also time stamps the image data so that the image data may be associated with times throughout the interventional procedure. The image data may also include additional meta-data, such as the C- arm pose, table position, and patient information, for example.

[0031] The images acquired by the medical imaging device 130 generally show the vasculature in the ROI 156, any interventional devices that may be located within the ROI 156 during the interventional procedure, landmarks on the vasculature and / or the interventional devices, and contrast agent that is present at the time of the image. As mentioned above, examples of interventional devices that might appear in the images include surgical instruments, catheters, biopsy tools, implants, and ultrasound transducers. Due to blood flow, the contrast agent washes away over time following injection, so the cumulative amount of contrast agentgenerally increases with each introduction into the vasculature.

[0032] Contrast estimation module 122 includes a first model (or algorithm) that estimates amounts of contrast agent within the vasculature throughout the interventional procedure. The first model may be a first machine learning model that has been previously trained, using supervised or unsupervised learning, based on features of previous image data associated with known amounts of contrast agents, as discussed below. The first machine learning model inputs at least image data from the image data module 121, and outputs estimated amounts of contrast agent present within the ROI 156 at interim times during the interventional procedure and / or an estimated total amount of contrast agent present within the ROI 156 over the course of the entire interventional procedure. In an embodiment, the contrast agent injector may be an automatic contrast agent injector. In this case, the estimated amounts of contrast agent present within the ROI 156 at interim times may be used to enhance the measurements made by the automatic contrast injector.

[0033] In various embodiments, the first machine learning model may also optionally input additional data, one or more of which may be used together with the image data to provide a richer input representation of the procedure and to enhance inference of the first machine learning model. The additional data may include contrast agent data about the type and concentrations of contrast agent, injector data about the type of contrast agent injector 140, patient information about the patient 150 (e.g., height, weight, age, and medical history), user data about the user 152 (e.g., identification, experience, past interventional procedures), procedure report data detailing previous interventional procedures on the patient 150, and / or meta data indicating positions and orientations of any movable portions of the medical imaging device 130. As an example, the injector data may inform the first machine learning model of the maximum amount of contrast injection per syringe fill. The injector data may identify the type of syringe indicating minimum and / or maximum pressure that may be applied during injection, which may inform the first machine learning model about the flow of the contrast. The patient information may also be used to indicate the type of procedure, allowing the first machine learning model to make more accurate inferences based on historical data for similar procedures that use similar amounts of contrast.

[0034] The additional data may further include video and / or logged events. Video data isprovided from the patient site, and may capture the hands of the user and the manual contrast agent injector 140. Using image processing algorithms, events in which the contrast agent injector 140 was used can be identified. Similarly, the hand movements of the user 152 captured by the video data may show the amount of fluid pushed into the vasculature of the patient 150 by operation of the contrast agent injector 140. The logged events may capture all or some of the above information transcribed by the user 152 and / or the operating room staff, including the corresponding times. Accordingly, estimating the amount of the contrast agent used during the interventional procedure may be further based at least in part on the logged events.

[0035] The first machine learning model may be a neural network model, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) neural network, or a transformer neural network, for example. The first machine learning model is trained, using the processor 112 for example, on training data before being implemented for a procedure. The training data may be historical data (actual and / or simulated) that includes previous image data and known amounts of contrast agent respectively indicated by the previous image data from previous interventional procedures. The historical data may be received from the database 128 and / or some other data source and / or data simulator, as would be apparent to one skilled in the art.

[0036] To estimate quantifiable insights from the training data, the first machine learning model must be trained on representative datasets of historical data, as discussed above. For instance, to estimate the total amount of contrast agent used in an interventional procedure, the first machine learning model may be trained on image data from image sequences (e.g., DSA sequences) that were generated using automated contrast injector pumps, which track amounts of contrast agent injected. Therefore, ground truth amount of contrast agent injected is known from the contrast injector pump, and the first machine learning model is trained to associate image features from the image sequence (e.g., multiple frames of images) with the known amounts of contrast agent. The image features may include size and brightness of pixels, and movement of pixels representing contrast with the ground truth amount of contrast agent. The pixels representing the contrast agent may be dark pixels or bright pixels, depending on the set up of the first machine learning model.

[0037] As mentioned above, the training may include supervised learning or unsupervisedlearning. The supervised learning is performed using annotations starting with identifying in each image the pixels indicating contrast agent. Subsequently, the model uses the image pixels with or without the segmented pixels to predict the amount of the contrast in the current image as well as the total amount of contrast administered. For supervised learning, the first machine learning model may use the annotations to compare predictions of amounts of contrast agent with ground truth. Alternatively, the first machine learning model may use the annotations to learn latent representations of contrast agent, such that representations from different classes or with different labels are maximally separated using contrastive learning. The annotations may be manually by users, or may be automatically generated based on an automatic injector. In the case of latter, the training of the first machine learning model is performed on data from an automatic injector, and the inference is performed on procedures with a manual injector. The unsupervised learning is performed without annotations. For unsupervised learning, the first machine learning model may be trained on a large population of training data from other users, and subsequently re-trained and fine-tuned on training data specific to the user 152 in order to provide personalized measurements. The first machine learning model learns from difference images or deformation maps across the image sequence to estimate flow of the contrast agent by unsupervised learning. In either case of supervised or unsupervised learning, the trained first machine learning model enables estimation of the amount of contrast used during the interventional procedure.

[0038] Inference of the first machine learning model includes applying image data from medical images acquired in real-time to the first machine learning model, and outputting the estimated amount of contrast agent present within the ROI 156 during the interventional procedure, as discussed above. The estimated amount of contrast agent may include estimated interim amounts of contrast agent present at interim times (or stages) during interventional procedure, the estimated cumulative amount of contrast agent present at each interim time, and / or the total cumulative amount of contrast agent present within the ROI 156 over the course of the entire interventional procedure. Use of the first machine learning model according to the embodiments described herein is particularly useful when automated contrast injector pumps are not used, and the contrast agent is injected manually, as discussed above. However, in an embodiment, the trained first machine learning model may be used with the automated contrastinjector pumps to enhance their measurements. For example, the contrast injector pumps mix contrast agent with various amounts of saline, which may make it difficult to determine exactly the amount contrast agent is injected in the patient. Also, the contrast agent may leak from valves, such that it does not inject entirely in the patient. Therefore, the information provided by the contrast injector pump may not be sufficient to determine the exact amount of contrast agent injected into patient. Applying the trained first machine learning model that uses imaging to estimate the amount of contrast agent may be used to enhance the information provided by the contrast injector pumps.

[0039] Usage characterization module 123 includes a second model that characterizes usage of the contrast agent, as estimated by the contrast estimation module 122. The usage characterization may be specific to the user 152 and / or the type of interventional procedure being performed by the user, and is based at least in part on the estimated amount of contrast agent used during the interventional procedure.

[0040] In addition, the second model may determine a usage trend over time based on the characterized usage of the contrast agent, together with previous characterized usages of the contrast agent from one or more previous interventional procedures by the same user 152. In this case, the second module may also score performance of the user 152 based on the usage trend and report the scoring of the performance to the user 152 via the display 116. The scoring may be subjective based on comparisons to previous performances by the same user 152, indicating whether the performance is better or worse than the user’s average score, for example. Alternatively, or in addition, the scoring may be more objective based on comparisons to previous performances by different users, again indicating whether the performance is better or worse than an average score of the different users, for example. In this context, the second model may also identify key improvements that may be made. For example, if the usage trend shows that the user 152 tends to use more contrast agent than average early on during interventional procedures, the second model may formulate a suggestion to the effect that the amount of contrast agent be reduced during the earlier stages of the interventional procedure, which suggestion may be displayed on the display 116. The user 152 may turn the suggestions feature on / off as per their preference.

[0041] The usage characterization module 123 may separate the usage characterizations bythe types of interventional procedure for which the contrast agent is used, so that the usage characterizations are procedure-specific. For example, the use of contrast agent during an aneurysm treatment procedure is different from the use of contrast agent during a stroke treatment procedure. Therefore, comparing these usages to each other may not provide meaningful insights for purposes of usage characterization.

[0042] The second model may be a statistical model or a second machine learning model that has been previously trained based on training data from previous interventional procedures of the same type, using supervised or unsupervised learning, as discussed below. The usage characterization module 123 provides the characterized usage to the display 116 for reporting the usage characterization to the user 152, as discussed below.

[0043] In an embodiment, the second model characterizes the usage of the contrast agent by determining whether estimated interim amounts of the contrast agent are above or below corresponding historical average interim amounts of contrast agent at the same interim times. In particular, the second model inputs the estimated interim amounts of the contrast agent at the interim times of the interventional procedure, respectively, as determined by the first machine learning model. The second model then associates the estimated interim amounts of the contrast agent with corresponding average interim amounts of contrast agent at corresponding interim times as determined for the previous interventional procedures, and determines whether the estimated interim amounts of the contrast agent are above or below the corresponding average interim amounts of contrast agent associated with the estimated interim amounts. In this case, the estimated interim amounts may be sorted into one of two clusters indicating above and below average usage, and displayed on the display 116. The same analysis and characterization may be performed for the estimated total amount of the contrast agent present within the ROI 156 over the course of the entire interventional procedure, as determined by the first machine learning model.

[0044] In another embodiment, which is more granular, the second model characterizes the usage of the contrast agent by sorting the estimated interim amounts of the contrast agent at the interim times into bins (classes) representing contrast agent amounts. For example, the bins may include 0-2mL / kg of contrast agent, 2-4mL / kg of contrast agent, 4-6mL / kg of contrast agent, and greater than 56mL / kg of contrast agent for each interim time, although other ranges and / ornumbers of bins may be incorporated without departing from the scope of the present teachings. In particular, the second model inputs the estimated interim amounts of the contrast agent at the interim times of the interventional procedure, respectively, as determined by the first machine learning model. The second model then associates the estimated interim amounts of the contrast agent with the multiple bins representing different contrast agent amounts as determined for the previous interventional procedures, and determines a distribution of the estimated amount of the contrast agent based on the associating of the estimated interim amounts of the contrast agent with the multiple bins using the second model. In this case, the estimated interim amounts may be sorted into multiple clusters corresponding to the multiple bins, and displayed on the display 116. The same analysis and characterization may be performed for the estimated total amount of the contrast agent present within the ROI 156 over the course of the entire interventional procedure, as determined by the first machine learning model.

[0045] When the second model is a second machine learning model, the second machine learning model may be a neural network model, such as a CNN, an RNN, an LSTM neural network, or a transformer neural network, for example. Similar to the discussion above, the second machine learning model is trained, using the processor 112 for example, before being implemented for a procedure. The second machine learning model may be trained using historical data (actual and / or simulated) with or without supervision including previous amounts of contrast agent used by the user 152 for previous interventional procedures, previous amounts of contrast agent used by other users for previous interventional procedures, types of interventional procedures, and types and concentrations of contrast agent. The historical data may be received from the database 128 and / or some other data source and / or data simulator, as would be apparent to one skilled in the art. Comprises a previously trained neural network model.

[0046] Similar to the first machine learning model, the second machine learning model must be trained on representative datasets of historical data. For instance, the second machine learning model may be trained on training data providing or enabling determination of average and / or binned interim amounts of contrast agent at interim times during similar interventional and / or total amounts of contrast agent during similar interventional procedures. As mentioned above, the training may include supervised learning using annotations or unsupervised learning withoutannotations. For example, the second machine learning model may be trained using clustering. In this approach, the second machine learning model is trained to cluster data in its latent representation (e.g., using autoencoders or variational autoencoder architectures). Then, a small sample of data from each cluster may be labeled by a user. For example, a cluster may be labeled as high contrast usage and another cluster may be labeled as low contrast usage. Such labels will be assigned to each cluster.

[0047] Inference of the second machine learning model includes applying the interim amounts and / or total amounts of contrast agent from the first machine learning model to the second machine learning model, and outputting the characterization of the interim amounts and / or total amounts of contrast agent, as discussed above. When the second machine learning model has been trained using clustering, during inference, incoming data will go through the second machine learning model and end up located within or closest to one of the previously defined clusters. The output prediction for that given incoming data will be the label of the cluster. The second machine learning model may further determine and output a usage trend over time by aggregating the current characterized usage of the contrast agent with previous characterized usages of the contrast agent for the same user 152.

[0048] Since data privacy is important with regard to patients and providers, federated learning may be used for training the first and / or second machine learning models in order to leverage larger datasets from different users and / or different healthcare facilities (e.g., hospitals). For example, the users may opt-in to share weights of the first and second machine learning models trained locally using their own data. The shared weights are used to train global first and / or second machine learning models, together with weights from various other users and healthcare facilities, without ever sharing data from any of the individual users. The federated learning is used to build the first and / or second machine learning models from this larger training dataset, and the first and / or second machine learning models may then be fine-tuned locally by the users to their specific data. Also, the federated learning enables the user 152 to view comparisons against data in the larger dataset from other users, such that the privacy of all users may be protected. The comparisons enable the user 152 to be informed as to whether they are more or less proficient than other users, for example.

[0049] Reporting module 124 of the memory 120 is configured to collect results from thecontrast estimation module 122 and the usage characterization module 123 for display on the display 116 in order to report the results to the user 152. The reporting module 124 causes the display 116 to display the estimated total amount of the contrast agent and the estimated interim amounts of the contrast agent used at the corresponding interim times during the interventional procedure output the first machine learning algorithm. The reporting module 124 also causes the display 116 to display the characterized usage of the contrast agent specific the user and / or the interventional procedure output by the second model. The display of the characterized usage may include indications of estimated interim amounts of the contrast agent being clustered above or below average interim amounts, and / or being clustered within bins representing different contrast agent amounts. The display of the characterized usage may also include display of usage trends based on the characterized usage of the contrast agent, and / or performance scores based on the usage trends.

[0050] In an embodiment, the usage characterization module 123 provides various quantifiable user-specific insights, which may be displayed by the reporting module 124 on the display 116 (e.g., via a dashboard). For instance, the reporting module 124 may generate a daily view of insights that is displayed on the display 116 to enable the user 152 to compare performances over several days with regard to characterization of the contrast agent usage, and to see, for example, whether they have used more or less contrast agent than usual. By keeping insights user specific, the privacy of user information is preserved. Also, when one or more components in the workflow of the interventional procedure is performed substantially differently compared to the historic behavior of the user 152, as determined by the second model, the usage characterization module 123 may trigger a notification to that effect, which is displayed on the display 116. The user 152 may also be given the choice to include or exclude data from the one or more components in the overall analysis.

[0051] In an embodiment, the reporting module 124 may compare the estimated interim amounts of the contrast agent with a threshold in real-time during the interventional procedure. The threshold indicates a predetermined maximum allowable amount of contrast agent to be used in that interventional procedure. When any of the estimated interim amounts of the contrast agent exceed the threshold, the reporting module 124 outputs an alert to the user 152 via the display 116, notifying the user 152 not to inject any more contrast agent for the remainder of theinterventional procedure, thereby protecting the patient 150.

[0052] The display 116 may show data enabling the user 152 to evaluate their performance over time. For example, the display 116 may display a daily dashboard that shows contrast agent usage per day and / or per procedure, with or without any comparisons across days and / or procedures. For instance, FIG. 2 is an example of a display showing analysis of contrast agent usage by a user, according to a representative embodiment. For purposes of illustration, the display is provided on a smart phone, although it is understood that the display may be provided on any form of display 116 as described above, without departing from the scope of the present teachings.

[0053] Referring to FIG. 2, display 200 shows a bar graph with vertical bars corresponding to different amounts of the contrast agent volume. Each bar is divided into contrast agent usage at different phases of the procedure (e.g., navigation, treatment, puffs, diagnostics, etc.), indicated by different shading. The bars are further separated by the types of procedure, which include arteriovenous malformations (A VMS), aneurysm and stroke in the depicted example.

[0054] FIG. 3 is a flow diagram of a method of estimating contrast agent usage by a user during an interventional procedure on a patient, according to a representative embodiment. The method depicted in FIG. 3 may be implemented at least in part by the processor 112 of the control unit 110 executing instructions stored in the memory 120, for example.

[0055] Referring to FIG. 3, the method includes receiving image data from a series of images acquired by a medical imaging device during the interventional procedure in block S311. The series of images may show a portion of vasculature of the patient through which the contrast agent flows and / or may show one or more interventional devices within the portion of the vasculature.

[0056] In block S312, the received image data is applied to a first machine learning model that has been previously trained based on features of previous image data associated with known amounts of contrast agents. The first machine learning model may be a neural network model, such as a CNN, an RNN, an LSTM neural network, or a transformer neural network, for example.

[0057] In block S313, a total amount of the contrast agent used during the interventional procedure is estimated by the first machine learning model based on similarity of features of thereceived image data and the features of the previous image data.

[0058] In block S314, interim amounts of the contrast agent used at corresponding interim times (stages) during the interventional procedure are optionally estimated (indicated by dashed box) by the first machine learning model based on similarity of the features of the received image data at the corresponding interim times and the features of the previous image data at the same interim times using the first machine learning model.

[0059] In block S315, the estimated amount(s) of contrast agent are reported to the user via a display. The reported estimated amount(s) of the contrast agent may include the estimated total amount of the contrast agent used during the interventional procedure and, optionally, the interim amounts of the contrast agent used at the corresponding interim times during the interventional procedure.

[0060] In block S316, usage of the contrast agent, specific to at least one of the user or the interventional procedure, is characterized based in part on the estimated total amount of contrast agent and / or the estimated interim amounts of the contrast agent used during the interventional procedure. The contrast agent usage is characterized using a second model, which may be a statistical model or second machine learning model that has been previously trained based on training data from previous interventional procedures of the same type, such as a CNN, an RNN, an LSTM neural network, or a transformer neural network, for example.

[0061] In block S317, the characterized usage of the contrast agent is reported to the user via the display. The characterized usage may include characterizing use of the contrast agent as above or below average, and / or characterizing use of the contrast agent as being within a bin of multiple predetermined bins having corresponding bin value ranges.

[0062] Although the present specification describes components and functions that may be implemented in particular embodiments with reference to particular standards and protocols, the disclosure is not limited to such standards and protocols. Such standards are periodically superseded by more efficient equivalents having essentially the same functions. Accordingly, replacement standards and protocols having the same or similar functions are considered equivalents thereof.

[0063] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are notintended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

[0064] One or more embodiments of the disclosure may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept. Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.

[0065] The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0066] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and theappended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description. 1

Claims

CLAIMS:

1. A method of estimating contrast agent usage by a user during an interventional procedure, the method comprising: receiving image data from a series of images acquired by a medical imaging device (130) during the interventional procedure (S311), wherein the series of images show a portion of a vasculature of the patient through which a contrast agent flows; applying the received image data to a first machine learning model (122) that has been previously trained based on features of previous image data associated with known amounts of contrast agents (S312); estimating, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data(S313); and reporting the estimated total amount of the contrast agent to the user.

2. The method of claim 1, wherein the features of the received image data include size and movement of dark or bright image pixels in the series of images, and wherein the dark or bright image pixels are associated with the contrast agent and landmarks on the images.

3. The method of claim 1, further comprising: estimating, by the first machine learning model, interim amounts of the contrast agent used at corresponding interim times during the interventional procedure based on similarity of the features of the received image data at the corresponding interim times and the features of the previous image data at the same interim times; and reporting the estimated interim amounts of the contrast agent to the user.

4. The method of claim 3, further comprising: applying at least one of the estimated total amount of contrast agent or the estimated interim amounts of the contrast agent used during the interventional procedure to a second model, wherein the second model comprises a statistical model or a second machine learningmodel that has been previously trained based on training data from previous interventional procedures of the same type, and characterizing, by the second model, usage of the contrast agent specific to at least one of the user or the interventional procedure; and reporting the characterized usage to the user.

5. The method of claim 4, wherein characterizing the usage the contrast agent comprises: associating, by the second model, the estimated interim amounts of the contrast agent with corresponding average interim amounts of contrast agent at corresponding interim times as determined for the previous interventional procedures; and determining, by the second model, whether the estimated interim amounts of the contrast agent are above or below average based on the associating of the estimated interim amounts of the contrast agent and the corresponding average interim amounts of the contrast agent.

6. The method of claim 4, wherein characterizing the usage of the contrast agent comprises: associating, by the second model, the estimated interim amounts of the contrast agent with multiple bins representing different contrast agent amounts as determined for the previous interventional procedures; and determining, by the second model, a distribution of the estimated interim amounts of the contrast agent based on the associating of the estimated interim amounts of the contrast agent with the multiple bins.

7. The method of claim 4, further comprising: determining a usage trend based on the characterized usage of the contrast agent and previous characterized usage of the contrast agent from at least one previous interventional procedure by the user.

8. The method of claim 7, further comprising: scoring performance of the user based on the usage trend; andreporting the scoring of the performance to the user.

9. The method of claim 3, further comprising: comparing, during the interventional procedure, the estimated interim amounts of the contrast agent with a threshold indicating a predetermined maximum allowable amount of contrast agent to be used in the interventional procedure; and outputting an alert to the user in real time when the estimated interim amounts of the contrast agent exceed the threshold.

10. The method of claim 1, further comprising: receiving video data and at least one logged event associated with the video data; and estimating the total amount of the contrast agent used during the interventional procedure further based on the at least one logged event.

11. A system for estimating contrast agent usage by a user during an interventional procedure on a patient (150), the system comprising: a medical imaging device (130) configured to acquire a series of images during the interventional procedure, wherein the series of images show a portion of vasculature of the patient through which contrast agent flows; a control unit (110) including at least one processor (112) and a non-transitory memory (120) storing instructions that, when executed, cause the at least one processor to: receive image data from the series of images acquired by the medical imaging device during the interventional procedure (S311); apply the received image data to a first machine learning model (122) that has been previously trained based on features of previous image data associated with known amounts of contrast agents (S312); estimate, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data (S313); and report the estimated total amount of the contrast agent to the user (S315).

12. The system of claim 11, wherein the first machine learning model comprises a neural network model.

13. The system of claim 11, wherein the instructions further cause the at least one processor to: estimate, by the first machine learning model, interim amounts of the contrast agent used at corresponding interim times during the interventional procedure based on similarity of the features of the received image data at the corresponding interim times and the features of the previous image data at the same interim times, and report the estimated interim amounts of the contrast agent to the user.

14. The system of claim 13, wherein the instructions further cause the at least one processor to: compare, during the interventional procedure, the estimated interim amounts of the contrast agent with a threshold indicating a predetermined maximum allowable amount of contrast agent to be used in the interventional procedure; and output an alert to the user in real time when the estimated interim amounts of the contrast agent exceeds the threshold.

15. The system of claim 13, wherein the instructions further cause the at least one processor to: characterize, by a second model, usage of the contrast agent specific to at least one of the user or the interventional procedure, based on at least one of the estimated total amount of contrast agent or the estimated interim amounts of the contrast agent used during the interventional procedure, wherein the second model comprises a statistical model or second machine learning model that has been previously trained based on training data from previous interventional procedures of the same type, wherein report the characterized usage to the user.

16. The system of claim 15, wherein the instructions cause the at least one processor to characterize the usage the contrast agent by: associating, by the second model, the estimated interim amounts of the contrast agent with corresponding average interim amounts of contrast agent at corresponding interim times as determined for the previous interventional procedures; and determining, by the second model, whether the estimated interim amounts of the contrast agent are above or below average based on the associating of the estimated interim amounts of the contrast agent and the corresponding average interim amounts of the contrast agent.

17. The system of claim 15, wherein the instructions cause the at least one processor to characterize the usage the contrast agent by: associating, by the second model, the estimated interim amounts of the contrast agent with multiple bins representing different contrast agent amounts as determined for the previous interventional procedures; and determining, by the second model, a distribution of the estimated interim amounts of the contrast agent based on the associating of the estimated interim amounts of the contrast agent with the multiple bins.

18. The system of claim 15, wherein the instructions cause the at least one processor to: determine a usage trend based on the characterized usage of the contrast agent and previous characterized usage of the contrast agent from at least one previous interventional procedure by the user.

19. The system of claim 15, wherein the second machine learning model comprises a previously trained neural network model.

20. A computer readable medium (120) storing instructions for estimating contrast agent usage by a user during an interventional procedure on a patient, wherein when executed, the instructions cause at least one processor (112) to:receive image data from a series of images acquired by a medical imaging device (130) during the interventional procedure (S311), wherein the series of images show a portion of vasculature of the patient through which contrast agent flows; apply the received image data to a first machine learning model (122) that has been previously trained based on features of previous image data associated with known amounts of contrast agents (S312); estimate, by the first machine learning model, a total amount of the contrast agent used during the interventional procedure based on similarity of features of the received image data and the features of the previous image data (S313); and cause the estimated total amount of the contrast agent to be reported to the user (S315).

Images