Methods and systems to minimize radiation exposure while maintaining optimal image quality

Abstract

A patient procedure imaging system (200), comprising: a radiation source (270) configured to acquire images of a patient during a patient procedure; and a processor (220) configured to: obtain one or more images acquired at an initial predetermined frame rate, analyze the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure, adjust the initial predetermined frame rate to the determined minimum required frame rate, obtain new images at the adjusted frame rate, predict images when the determined minimum required frame rate is below a predetermined rate, and combine the new images at the adjusted frame rate and the predicted images to generate a complete image sequence.

Description

METHODS AND SYSTEMS TO MINIMIZE RADIATION EXPOSURE WHILE MAINTAINING OPTIMAL IMAGE QUALITYField of the Disclosure

[0001] The present disclosure is directed generally to methods and systems for minimizing hazardous radiation exposure during a patient procedure.Background

[0002] Patient exposure to harmful radiation is an unwanted side effect of many medical procedures. For example, exposure to X-ray for extended periods of time is a routine clinical practice during an endovascular procedure in which a catheter is inserted into patient’s blood vessels to deliver a treatment to a target site. This continuous acquisition of sequences of X-ray images is known as a fluoroscopy run. The amount of exposure is typically controlled by the operating room (OR) staff wearing protective garments, using X-ray absorbing shields between the source and the personnel, and / or staff turning the source on and off manually at various time points during the procedure. However, these protective measures often do not adequately protect the patient, and parts of the clinician’s bodies (such as their head or arms having to get close to the X-ray source) are not covered while handling the equipment around the table.

[0003] X-ray imaging is utilized during endovascular navigation in order to visualize the device that is being advanced inside the patient’s body, as well as to visualize the vessels in which the device is moving. This visualization allows the interventionalist to determine how to handle the catheter and guidewire to achieve a smooth and rapid navigation toward the target. These X-ray sources are, additionally, equipped with the capability to change the delivered radiation intensity or dose, and the X-ray frame rate (FR). The X-ray frame rate is a feature that determines with what frequency a field of view is imaged by X-ray to allow a full image reconstruction. For example, an FR equal to 1 frame per second (fps) would allow visualization of the guidewire tip once every second. Such features are utilized to adjust the amount of exposure depending on the situation. While the FR, dose, and other variables can be modified on the X-ray imaging system, this is currently a manual process and users typically do not adjust these values during a fluoroscopy run.

[0004] Recent technological advancements in endovascular device design and robotics have created new potentials for navigation without always requiring X-ray imaging. Some of these technologies include robotic systems that can provide information about how far and with what velocity a device has advanced into patient’s body, shape sensing devices (e.g., fiber-optic-based devices or technology) with the capability for 3D localization of hundreds of points along a catheter or guidewire, electromagnetic (EM) navigation, magnetically steerable devices, and so on. Despite the presence of such capabilities, there has not been sufficient progress in utilizing them for an intelligent image-based minimization of X-ray usage while maintaining sufficiently high image quality and resolution.Summary of the Disclosure

[0005] There is thus a continued unmet need for methods and systems that minimize harmful radiation exposure to patients and clinicians during a patient procedure, without negatively impacting the outcome of the procedure. Various embodiments and implementations are directed to a method and system for determining a minimal frame rate during a patient procedure using a radiation-based patient procedure imaging system. The system receives images during the procedure at an initial frame rate. The system determines a lower, minimum required frame rate for a subsequent window of the patient procedure. For example, a trained machine learning model such as a neural network may analyze the received images and determine the frame rate. The system adjusts the frame rate to the lower rate and receives one or more new images at that lower frame rate. The system provides the one or more images received at the adjusted frame rate to a clinician via a user interface.

[0006] According to an aspect, imaging system for performing a patient procedure is provided. The system includes: a processor configured to: obtain one or more images acquired at an initial predetermined frame rate; analyze the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure; adjust the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate; obtain one or more new images at the adjusted frame rate; predict one or more images when the determined minimum required frame rate is below a predetermined rate; and combine the one or more newimages at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

[0007] According to an embodiment, the system further includes a radiation source configured to acquire images of a patient during a patient procedure.

[0008] According to an embodiment, the processor is further configured to apply a trained machine learning model configured to determine the minimum required frame rate for the subsequent window of the patient procedure.

[0009] According to an embodiment, the machine learning model is trained to generate the one or more predicted images when the determined minimum required frame rate is below a predetermined rate.

[0010] According to an embodiment, the system further includes a user interface configured to provide the complete image sequence.

[0011] According to an embodiment, the processor is further configured to obtain additional navigation data from a navigation data source, and wherein analyzing the one or more images to determine the minimum required frame rate for a subsequent window of the patient procedure further comprises analyzing the received additional navigation data.

[0012] According to an embodiment, the navigation data source is a second imaging modality.

[0013] According to an embodiment, the additional navigation data is generated by the navigation source before the patient procedure.

[0014] According to an embodiment, the additional navigation data is generated by the navigation source during at least a portion of the patient procedure.

[0015] According to an embodiment, the system further includes a catheter utilized inside the patient during the patient procedure, and wherein the complete image sequence is utilized for navigation of the catheter.

[0016] According to an embodiment, another processor is configured to train a machine learning model to generate the trained machine learning model, the another processor configured to: obtain training data comprising imaging data from a plurality of patient procedures; train the machine learning model to determine a minimum required frame rate for a subsequent window of a patient procedure, wherein the minimum required frame rate minimizes the frame rate while maintaining an image quality necessary to successfully perform the patient procedure; and store the trained machine learning model in memory.

[0017] According to an embodiment, the imaging data is obtained at a frame rate at or above the initial predetermined frame rate.

[0018] According to an embodiment, to train the machine learning model, the another processor is further configured to identify, by the machine learning model, one or more factors within the training data influencing the minimum required frame rate.

[0019] According to an embodiment, the one or more factors comprises one or more of the patient procedure being performed, an anatomy of the patient, and a behavior of a device being navigated inside the patient including translational velocity or rotational velocity of the device.

[0020] According to another aspect is a method for performing a patient procedure. The method comprising: obtaining one or more images of a patient acquired at an initial predetermined frame rate; analyzing the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure; adjusting the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate; obtaining one or more new images at the adjusted frame rate; predicting one or more images when the determined minimum required frame rate is below a predetermined rate; and combining the one or more new images at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

[0021] According to an embodiment, the method further includes applying a trained machine learning model configured to determine the minimum required frame rate for the subsequent window of the patient procedure and to predict the one or more predicted images when the determined minimum required frame rate is below a predetermined rate.

[0022] According to an embodiment, the method further includes generating the trained machine learning model by: obtaining training data comprising imaging data from a plurality of patient procedures; training the machine learning model to determine a minimum required frame rate for a subsequent window of a patient procedure, wherein the minimum required frame rate minimizes the frame rate while maintaining an image quality necessary to successfully perform the patient procedure; and storing the trained machine learning model in memory.

[0023] According to an embodiment, the method includes training of the machine learning model comprises by identifying, by the machine learning model, one or more factors within the training data influencing the minimum required frame rate.

[0024] According to another aspect is a non-transitory computer-readable storage medium having stored a computer program comprising instructions. The instructions, when executed by a processor, cause the processor to obtain one or more images acquired at an initial predetermined frame rate; analyze the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure; adjust the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate; obtain one or more new images at the adjusted frame rate; predict one or more images when the determined minimum required frame rate is below a predetermined rate; and combine the one or more new images at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

[0025] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0026] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.Brief Description of the Drawings

[0027] In the drawings, like reference characters generally refer to the same parts throughout the different views. The figures showing features and ways of implementing various embodiments and are not to be construed as being limiting to other possible embodiments falling within the scope of the attached claims. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0028] FIG. 1 is a flowchart of a method for radiation-based imaging, in accordance with an embodiment.

[0029] FIG. 2 is a schematic representation of a patient procedure imaging system, in accordance with an embodiment.

[0030] FIG. 3 is a flowchart of a method for training a machine learning model, in accordance with an embodiment.

[0031] FIG. 4 is a flowchart of a method for training a machine learning model, in accordance with an embodiment.Detailed Description of Embodiments

[0032] The present disclosure describes various embodiments of a system and method configured to determine a minimal imaging frame rate during a patient procedure using a radiationbased patient procedure imaging system. More generally, Applicant has recognized and appreciated that it would be beneficial to provide a method and system to minimize harmful radiation exposure to patients and clinicians during a patient procedure, without negatively impacting the outcome of the procedure. A patient procedure imaging system receives images during the procedure at an initial frame rate. The system analyzes the received images to determine a lower, minimum required frame rate for a subsequent window of the patient procedure. In some embodiments, a trained machine learning model such as a neural network analyzes the received images to determine the lower, minimum required frame rate. The system adjusts the initial frame rate to the lower rate and receives one or more new images at that lower frame rate. The system further generates one or more predicted images when the determined minimum required frame rate is below a predetermined rate, and combines the received one or more images at the adjusted frame rate and the generated one or more predicted images to generate a complete image sequence. The system provides the images received at the adjusted frame rate, and / or the complete image sequence to a clinician via a user interface.

[0033] The embodiments and implementations disclosed or otherwise envisioned herein can be utilized with any system that may utilize or benefit from minimizing patient imaging and harmful radiation exposure. For example, one application of the embodiments and implementations disclosed or otherwise envisioned herein is minimizing X-ray imaging, such as during catheterization and other patient procedures. One application is to improve the functionality of thePhilips® Azurion® imaging system and software (manufactured by Koninklijke Philips, N.V.), among other products. However, the disclosure is not limited to these devices or systems, and thus disclosure and embodiments disclosed herein can encompass any system that may utilize or benefit from minimizing patient imaging and radiation exposure.

[0034] Referring to FIG. 1, in one embodiment, is a flowchart of a method 100 for radiationbased imaging using a patient procedure imaging system 200. The methods described in connection with the figures are provided as examples only, and shall be understood not to limit the scope of the disclosure. The patient procedure imaging system can be any of the systems described or otherwise envisioned herein. The patient procedure imaging system can be a single system or multiple different systems. According to one non-limiting example, method 100 is utilized for X- ray imaging, although other radiation-based imaging systems are possible.

[0035] At step 105 of the method, a patient procedure imaging system 200 is provided. Referring to an embodiment of a patient procedure imaging system 200 as depicted in FIG. 2, for example, the system comprises one or more of a processor 220, memory 230, user interface 240, communications interface 250, and storage 260, interconnected via one or more system buses 212. It will be understood that FIG. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated. Additionally, patient procedure imaging system 200 can be any of the systems described or otherwise envisioned herein. Other elements and components of the patient procedure imaging system 200 are disclosed and / or envisioned elsewhere herein.

[0036] According to an embodiment, the patient procedure imaging system 200 comprises or is in direct or indirect communication with a radiation source 270. The radiation source can be, for example, a radiation-based imaging device, machine, or system. The radiation-based imaging device may be any device designed or configured to obtain images at a plurality of different frame rates using radiation. For example, the radiation-based imaging device may be an X-ray device, component, or machine, and may be configured to obtain X-ray images at a plurality of different frame rates. Accordingly, the radiation-based imaging device 270 comprises one or more adjustable settings and / or parameters, including but not limited to an adjustable frame rate.

[0037] According to an embodiment, the patient procedure imaging system 200 comprises or is in direct or indirect communication with a second imaging modality 280. The second imagingmodality is configured to obtain images during a patient procedure. The second imaging modality may be any imaging device, and may obtain one or more images using any imaging modality. The most common forms of imaging modality are magnetic resonance imaging (MRI), ultrasound, computed tomography scan (CT scan), and nuclear imaging such as Positron Emission Tomography (PET), although many other types of health- or medicine-based imaging modalities are possible. The one or more images obtained using the imaging modality may be obtained from a patient or other individual.

[0038] According to an embodiment, the patient procedure imaging system 200 is utilized during a patient procedure to minimize X-ray exposure to patients and clinicians without negatively impacting the outcome of the procedure. According to one embodiment, the patient procedure is a procedure in which a catheter or other instrument is navigated within the body, although many other patient procedures are possible including procedures without a catheter or other instrument. In the case of navigation within the patient, the system uses available information, including information about the device location relative to the anatomy in which it is being navigated, its behavior such as translational and rotational velocity, and the anatomical features, to determine what X-ray frame rate is essential or desired at each time point. According to an embodiment, image features influences the optimal frame are learned and utilized to enable missing-frame replacement without compromising image quality.

[0039] Thus, the system can determine or estimate the minimum essential frame rate (MEFR) at each point in time according to at least one input from a catheter procedure (such as fluoro image features, robotics haptic feedback, etc.) and learn how to replace missing frames in the case of low frame rate to avoid any compromise of image resolution and quality. This system can optionally include a neural network that is trained on input data including high-FR X-ray images or fluoroscopy runs (e.g., 50-60 fps sequences) in which the device (e.g., catheter and / or guidewire) used for navigation is observable. If available, the path to the target site - which can be determined by fluoroscopy throughout the procedure - can also be used. Additional data from sources such as robotics, fiber optics, or any other device that can enable localization of the catheter without a need for X-ray, can be used as complementary information to further reduce the required FR for achieving a good image quality for device navigation. Thus, the system overcomes the lack of intelligence in interventional systems for automatic adjustment of X-ray framerate depending onthe procedure phase, anatomical complexity, and navigation difficulty, while maintaining a high image quality.

[0040] During a patient procedure, in accordance with one embodiment, data such as a series of X-ray images containing a moving device (such as catheter) is input into the system, and the system analyzes the relevant image sequence features to determine the MEFR (e.g., between 1 and 60 fps) required for smooth navigation while replacing the missing frames with high-quality images. The system can utilize a video frame extrapolation network to learn which features, and to what extent, influence the MEFR such the system maintains, for instance, a Structural Similarity Index (SSIM) above 0.95 when comparing predicted frames with ground truth frames available in input frame-grabbed data. Once trained, the model will then be able to let the live X-ray imaging system start with a high FR, determine (based on sequence / image features) what MEFR is appropriate for each next 1 second window (or any other time frame for the window) in the future, and dynamically apply that to the device FR settings, while replacing the missing frames to avoid any image quality degradation. Depending on the criticality of the procedure step, the window size may be smaller or larger than 1 second. Therefore, the system will automatically increase the FR at more challenging steps of navigation and decrease at less complex steps based on what it has experienced during the last few frames of the live imaging sequence while maintaining the original imaging resolution.

[0041] Accordingly, returning to method 100 in FIG. 1, a patient procedure is initiated. The patient procedure can be any procedure for which radiation is utilized for imaging or any other component of the procedure. According to one embodiment, the patient procedure is a procedure in which a catheter or other instrument is navigated within the body, although many other patient procedures are possible including procedures without a catheter or other instrument.

[0042] At step 110 of the method, the patient procedure imaging system 200 receives one or more images of a patient during the initiated patient procedure. The one or more images are obtained at an initial predetermined frame rate, which in many cases will be a high frame rate, such as a typical frame rate utilized by prior art systems. The images of the patient may be obtained by and thus received from radiation-based imaging device 270, which may be any device, component, or machine configured to obtain images at a plurality of different frame rates usingradiation. The radiation-based imaging device 270 may be a component of patient procedure imaging system 200, or it may be in wired and / or wireless communication with the system.

[0043] At optional step 120 of the method, the patient procedure imaging system 200 receives additional navigation data from a navigation data source. This additional navigation data may be obtained prior to initiation of the patient procedure, or may be obtained in part or in whole during the patient procedure. Thus, the additional navigation data may be obtained prior to initiation of the patient procedure and stored until that data is utilized during the patient procedure. Additionally or alternatively, the additional navigation data may be obtained in part or in whole during the patient procedure and may be stored for subsequent use or may be utilized by the system immediately. The additional navigation data may be any information that facilitates navigation of a catheter or other instrument is within the patient during the patient procedure.

[0044] According to an embodiment, the navigation data source is the second imaging modality (280), which may be any imaging device, and may obtain one or more images using any imaging modality. The second imaging modality may obtain images of the patient before and / or during the patient procedure.

[0045] According to an embodiment, the additional navigation data may be obtained from or for a robotically controlled device. The robotically controlled device can determine the length of a device that has been inserted into a patient, the approximate tip velocity, the torque applied to the device, and other parameters about the device. This information can thus comprise additional navigation data to be added to make the analysis less complex with regard to device shape, location, and velocity at each time point. This additional information lets the network be optimized at lower input / output ratios and thus will result in lower possible MEFR values for fluoroscopy runs at various stages of the procedure.

[0046] According to an embodiment, the additional navigation data may be obtained from or for shape sensed devices (e.g., fiber optic devices) and / or electromagnetic (EM) devices. The shape, tip location, tip velocity, and / or other information from these devices can be used as additional inputs for the system to further enhance its predictive capacity and decrease the requirement for increasing the input / output frame ratio. Such additional input can result in a near-zero MEFR, considering that, given a pre-planned vessel path for an area of the anatomy, the device locationand behavior can be fully reconstructed without an essential need for extensive X-ray-based visualization.

[0047] At step 130 of the method, the patient procedure imaging system 200 analyzes the received one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure. This analysis may be performed by the processor 220 of system 200. The analysis will provide, as output based on the received images and any other input information, an appropriate minimum required frame rate for images for a subsequent window of the patient procedure. For example, the subsequent window of the patient procedure might be the next 1 second of the procedure, or a longer or shorter subsequent window.

[0048] According to an embodiment, the analysis of the received one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure is performed by a trained machine learning model 262 of the system. The machine learning model can be any network, algorithm, classifier, or artificial intelligence component which is trained to utilize the described input and generate the described output.

[0049] According to an embodiment, the machine learning model is a neural network trained (such as according to the methods described or otherwise envisioned herein) to receive high-FR X-ray image data and output an appropriate MEFR according to a specific time window of a predetermined or adjustable length. The neural network can be optimized in a self-supervised manner using a perceptual loss function that compares extrapolated output frames with their corresponding ground truth input frames. According to one embodiment, the network can be a convolutional encoder-decoder that, in each batch, inputs 2-60 previous frame sequences, deconstructs their features into a low dimensional representation, and reconstructs the next 1 -60 frames (with a 2 / 60 ratio of input / output in the beginning) from the low dimensional representation. The length of these sequences is only an example, and other sequence lengths are possible. The loss and SSIM can then be calculated using the extrapolated frames compared with the input frames. This network can, then, be wrapped in an architecture similar to that of automated hyperparameter optimization algorithms, where if training loss cannot converge to a value lower than a threshold after a certain number of training epochs, the training for that region of the image / sequence (i.e., a certain procedure step) will be restarted with an increased input / output frame ratio (i.e., increased MEFR) to reduce the complexity of the problem and achieve anacceptable loss value. The wrapper network can, additionally, include a convolutional classification network that learns the image / sequence features at each temporal region (i.e., each procedure step), including device behavior (e.g., device tip moving back and forth near a vessel branch for an extended period), that influence the changes in MEFR, as described above, and outputs optimized weights that define this relationship. The optimized classification model, on its own, can then be used to infer a proper MEFR for each temporal region without necessarily requiring the aforementioned frame extrapolation (or frame synthesis) network.

[0050] According to an embodiment, the determination by the system of a minimum required frame rate for a subsequent window of the patient procedure is further based on an analysis of the additional navigation data received from the navigation data source. For example, processor 220 of system 200 may utilize both the received one or more images and the received additional navigation data to determine the minimum required frame rate for images for a subsequent window of the patient procedure. As set forth herein, the additional navigation data received from the navigation data source can be any data that can be utilized by the processor to determine the minimum required frame rate. For example, the additional navigation data can be imaging data, information about the device(s) utilized in the patient procedure such as catheter information, and / or any other information.

[0051] According to one embodiment, the image reconstruction loss is used directly at inference to compute a change in FR based on an experimentally-derived relationship between the two. If the loss is small, the change in FR may be large (approaching zero starting from an initial maximum FR value), whereas if the loss is large the change in FR may be smaller. As long as the loss is below a threshold, FR is reduced. If the loss goes above threshold, FR is increased to control the loss. In this embodiment, the change in FR will always be slower than real time (i.e., 1 second behind if predicting frames 1 second in the future). This embodiment can be used at certain framerates such that the change in FR is faster than human reaction.

[0052] According to one embodiment, when multiple C-arm views are available for X-ray imaging, as in a biplane X-ray imaging system, the system may use image sequences from both views to compute the minimum required frame rate. The neural network may be adapted to a Siamese network architecture to accommodate two streams of input, and can output a single minimum required frame rate. If the FR of image acquisition can be modulated individually on thetwo arms of the biplane system, then the two inputs may be processed in parallel through the network, and output two separate minimum required frame rate which are applied to the corresponding arm of the biplane system. This may be applicable in a situation where the distal end of the device is foreshortened in one view and the changes in its shape are not seen in that view. The frame rate for that view can remain low, while the frame rate in the other view may be increased if the proximal end of the device is more clearly visible and increasing the frame rate may ease the process of cannulating a vessel, for example. This can significantly reduce the radiation burden of biplane systems.

[0053] At step 140 of the method, the patient procedure imaging system 200 adjusts the initial frame rate to the determined minimum required frame rate, for at least the duration of the subsequent window of the patient procedure. This can comprise, for example, the system sending a command to the radiation source 270 to adjust the frame rate to the determined minimum required frame. The command can be sent to the radiation source, such as an X-ray machine, which can be a component of system 200 or a component in communication with system 200, via wired and / or wireless communication. This can be a fully automated process such that it can be performed very quickly and efficiently, including on a second-by-second basis (or optionally even faster). According to another embodiment, the command is sent to a clinician, such as via a user interface of the system, and the clinician subsequently adjusts the frame rate of the radiation source.

[0054] At step 150 of the method, the patient procedure imaging system 200 receives from radiation-based imaging device 270 one or more images of a patient during the initiated patient procedure, at the adjusted frame rate. The images may be utilized immediately, or they may be temporarily or permanently stored for subsequent use by the system. Additionally, these new images may be fed back into the system to determine a minimum required frame rate for a subsequent window of the patient procedure.

[0055] At step 160 of the method, the patient procedure imaging system 200 determines that the frame rate is below a predetermined frame rate, and generates or extrapolates one or more predicted images. The predetermined frame rate may be any frame rate that the system or a clinician determines jeopardizes the quality of the imaging obtained by the system. The predetermined frame rate may be determined experimentally, may be determined by a set of rules, or may be determined by a clinician, among other mechanism. For example, system 200 maycomprise a user interface into which the clinician enters a predetermined frame rate, and / or system 200 may be preprogrammed with a specific frame rate that comprises the predetermined frame rate, and / or may be preprogrammed with a set of rules that determines the predetermined frame rate.

[0056] According to an embodiment, the predetermined frame rate may be zero. Extrapolated image frames for one or more time points can be generated or utilized when the radiation source is off or otherwise not gathering images. For example, when the radiation source is off or otherwise not gathering images such as for a fraction of a second - such as between 1 / 60 - 59 / 60 of a second - extrapolated image frames are generated or utilized.

[0057] According to an embodiment, deep learning-based video frame interpolation and extrapolation methods are utilized to generate extrapolated X-ray frames. Accordingly, the trained machine learning model 262 of the system, which can be any model, network, algorithm, classifier, or artificial intelligence component, can be trained to extrapolate one or more X-ray frames for one or more time points, using input imaging obtained by the system.

[0058] At step 170 of the method, the patient procedure imaging system 200 combines one or more images received at the adjusted frame rate with one or more of the extrapolated or predicted images in order to generate a complete image sequence. “Combine” may mean, for example, inserting one or more of the extrapolated or predicted images between two of the images received at the adjusted frame rate to generate an image sequence comprising both the received and generated images. Thus, the patient procedure imaging system 200 may comprise an algorithm, software, or other video processing module configured to combine received and generated images into a complete image sequence. Once generated, the complete image sequence can be utilized immediately, and / or it may be temporarily or permanently stored in local and / or remote memory for future use.

[0059] At step 180 of the method, the patient procedure imaging system 200 provides the complete image sequence to a user, such as a clinician, via a user interface 240. The complete image sequence may be provided via the user interface using any method for displaying imaging. The system may also provide other information via the user interface, including but not limited information about the patient, the patient procedure, the determined minimum required frame rate, and / or information about the received and generated images within the complete image sequence.

[0060] Referring to FIG. 3, in one embodiment, is a flowchart of a method 300 for training the machine learning model 262 of the patient procedure imaging system 200. This method may be performed by the patient procedure imaging system, or may be performed by another system such as a machine learning model training system.

[0061] At step 310 of the method, the training system receives training data comprising imaging data for a plurality of patient procedures. The training data may comprise, for example, historical image data, such as fluoroscopy procedures, obtained from a large number of procedures. The historical image data may comprise static or moving devices (e.g., catheter, micro-catheter, guidewire, or any other device); (ii) static or moving background anatomy (including the planned path to the target site); and / or (ii) segmentation maps of devices and vessels per frame, if available. The training data may also comprise other information. This training data may be curated by an expert such as a clinician, or it may be obtained and utilized without curation. The training data may be received from any source. For example, the training data may be received from a database or other component of the patient procedure imaging system or a training system. According to an embodiment, the patient procedure imaging system 200 comprises or is in direct or indirect communication with an imaging database which comprises some or all of the training data set.

[0062] According to an embodiment, the training system may comprise a data pre-processor or similar component or algorithm configured to process the received training data. For example, the data pre-processor analyzes the training data to remove noise, bias, errors, and other potential issues. The data pre-processor may also analyze the input data to remove low quality data. Many other forms of data pre-processing or data point identification and / or extraction are possible.

[0063] At step 320 of the method, the training system trains the machine learning model, using the training data, to determine a minimum frame rate that maintains a required image quality. The network can be any algorithm, classifier, or model capable of creating the output, including but not limited to machine learning algorithms, classifiers, and other algorithms. The machine learning model is trained using any method for training a machine learning algorithm. The trained machine learning model is a unique algorithm based on the training data used to train the algorithm. Following training, the system comprises a trained machine learning model 262.

[0064] According to an embodiment, the machine learning model is a neural network trained to receive high frame rate X-ray image data and output an appropriate MEFR for a subsequentwindow, optimized in a self-supervised manner using a perceptual loss function that compares extrapolated output frames with their corresponding ground truth input frames. The neural network can be a convolutional encoder-decoder that, in each batch, inputs 2-60 previous frame sequences, deconstructs their features into a low dimensional representation, and reconstructs the next 1 -60 frames (with a 2 / 60 ratio of input / output in the beginning) from the low dimensional representation. Other sequence lengths can also be used. The loss and SSIM can be calculated using the extrapolated frames compared with the input frames. This network can be wrapped in an architecture similar to that of automated hyperparameter optimization algorithms, where if training loss cannot converge to a value lower than a threshold after a certain number of training epochs, the training for that region of the image / sequence (i.e., a certain procedure step) will be restarted with an increased input / output frame ratio (i.e., increased MEFR) to reduce the complexity of the problem and achieve an acceptable loss value. The wrapper network can, additionally, include a convolutional classification network that learns the image / sequence features at each temporal region (i.e., each procedure step), including device behavior (e.g., device tip moving back and forth near a vessel branch for an extended period), that influence the changes in MEFR, as described herein, and outputs optimized weights that define this relationship. The optimized classification model, on its own, can then be used to infer a proper MEFR for each temporal region without necessarily requiring the aforementioned frame extrapolation (or frame synthesis) network.

[0065] According to an embodiment, therefore, the neural network is trained to generate an output comprising the determined minimum required frame rate for each region (such as 1 second clips) of a sequence that in a live X-ray imaging case will be dynamically changing the X-ray FR. The neural network is further trained to generate and provide the extrapolated X-ray frames for each time point when the determined minimum required frame rate is lower than a maximum (i.e., the X-ray is off for a fraction of a second, for example between 1 / 60 - 59 / 60 of a second). This fills in the live X-ray sequence with an accuracy that is optimized to be nearly equal to that of the original image series.

[0066] According to an embodiment, referring to FIG. 4, the machine learning model is configured to be trained in a fully or semi-supervised manner, shown as flowchart 400. This embodiment might require additional clinical input as annotations on frame-grabbed image sequences of endovascular procedures. Annotations for this task could include experts such as clinicians labeling per temporal region along a procedure what FR (e.g., between 1-60 fps) wouldbe essential to be able to smoothly navigate, based on clinical experience. The machine learning model will then be simplified to the classification wrapper described herein in which the image sequences can be fed as input and the physician annotations can be used as ground truth for the model to learn what image / sequence features change the MEFR to what extent.

[0067] At step 330 of the method, the trained machine learning model 262 is stored for future use. According to an embodiment, the trained machine learning model 262 may be stored in local or remote storage.

[0068] Referring to FIG. 2 is a schematic representation of a patient procedure imaging system 200. System 200 may be any of the systems described or otherwise envisioned herein, and may comprise any of the components described or otherwise envisioned herein. It will be understood that FIG. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated.

[0069] According to an embodiment, system 200 comprises a processor 220 capable of executing instructions stored in memory 230 or storage 260 or otherwise processing data to, for example, perform one or more steps of the method. Processor 220 may be formed of one or multiple modules. Processor 220 may take any suitable form, including but not limited to a microprocessor, microcontroller, multiple microcontrollers, circuitry, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a single processor, or plural processors.

[0070] Memory 230 can take any suitable form, including a non-volatile memory and / or RAM. The memory 230 may include various memories such as, for example LI, L2, or L3 cache or system memory. As such, the memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices. The memory can store, among other things, an operating system. The RAM is used by the processor for the temporary storage of data. According to an embodiment, an operating system may contain code which, when executed by the processor, controls operation of one or more components of system 200. It will be apparent that, in embodiments where the processor implements one or more of the functions described herein in hardware, the software described as corresponding to such functionality in other embodiments may be omitted.

[0071] User interface 240 may include one or more devices for enabling communication with a user. The user interface can be any device or system that allows information to be conveyed and / or received, and may include a display, a mouse, and / or a keyboard for receiving user commands. In some embodiments, user interface 240 may include a command line interface or graphical user interface that may be presented to a remote terminal via communication interface 250. The user interface may be located with one or more other components of the system, or may located remote from the system and in communication via a wired and / or wireless communications network.

[0072] Communication interface 250 may include one or more devices for enabling communication with other hardware devices. For example, communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. Additionally, communication interface 250 may implement a TCP / IP stack for communication according to the TCP / IP protocols. Various alternative or additional hardware or configurations for communication interface 250 will be apparent.

[0073] Storage 260 may include one or more machine-readable storage media such as readonly memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media. In various embodiments, storage 260 may store instructions for execution by processor 220 or data upon which processor 220 may operate. For example, storage 260 may store an operating system 261 for controlling various operations of system 200.

[0074] It will be apparent that various information described as stored in storage 260 may be additionally or alternatively stored in memory 230. In this respect, memory 230 may also be considered to constitute a storage device and storage 260 may be considered a memory. Various other arrangements will be apparent. Further, memory 230 and storage 260 may both be considered to be non-transitory machine-readable media. As used herein, the term non-transitory will be understood to exclude transitory signals but to include all forms of storage, including both volatile and non-volatile memories.

[0075] While system 200 is shown as including one of each described component, the various components may be duplicated in various embodiments. For example, processor 220 may include multiple microprocessors that are configured to independently execute the methods describedherein or are configured to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein. Further, where one or more components of system 200 is implemented in a cloud computing system, the various hardware components may belong to separate physical systems. For example, processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0076] According to an embodiment, system 200 comprises or is in direct or indirect communication with a radiation source 270, which may be a radiation-based imaging device, machine, or system . The radiation-based imaging device may be any radiation-based imaging device, component, or machine configured to obtain images at a plurality of different frame rates using radiation. The radiation-based imaging device 270 comprises one or more adjustable settings and / or parameters, including but not limited to an adjustable frame rate. According to one nonlimiting embodiment, the radiation source 270 is an X-ray machine or system.

[0077] According to an embodiment, system 200 comprises or is in direct or indirect communication with a second imaging modality 280. The second imaging modality is configured to obtain images during a patient procedure. The second imaging modality may be any imaging device, and may obtain one or more images using any imaging modality. The most common forms of imaging modality are magnetic resonance imaging (MRI), ultrasound, computed tomography scan (CT scan), and nuclear imaging such as Positron Emission Tomography (PET), although many other types of health- or medicine-based imaging modalities are possible.

[0078] According to an embodiment, storage 260 of system 200 may store one or more algorithms, modules, and / or instructions to carry out one or more functions or steps of the methods described or otherwise envisioned herein. For example, storage 260 may comprise, among other instructions or data, a trained machine learning model 262, training instructions 263, and / or reporting instructions 264.

[0079] According to an embodiment, the trained machine learning model 262 of the patient procedure imaging system 200 is trained to analyze one or more received images to determine a minimum required frame rate for a subsequent window of the patient procedure, to maintain image quality. The machine learning model is also trained to extrapolate images and to insert the extrapolated images into obtained images when the frame rate is below a predetermined frame rate.The machine learning model can be any model, network, algorithm, classifier, or artificial intelligence component which is trained to utilize the described input and generate the described output.

[0080] According to an embodiment, the machine learning model is a neural network trained (such as according to the methods described or otherwise envisioned herein) to receive high-FR X-ray image data and output an appropriate MEFR according to a specific time window of a predetermined or adjustable length. According to an embodiment, the machine learning model is also trained to analyze received additional navigation data together with the one or more received images in order to determine the minimum required frame rate for a subsequent window of the patient procedure. The trained machine learning model is unique based on the training data used to train the machine learning model. Once generated, the trained machine learning model 262 can be utilized immediately, or it may be stored in local and / or remote memory for future use.

[0081] According to an embodiment, training instructions 263 direct the system to train a machine learning model 262 of the patient procedure imaging system 200. The instructions direct the system to: at step 310 of the method 300 in FIG. 3, for example, retrieve, obtain, or receive training data comprising imaging data for a plurality of patient procedures. The training data may comprise, for example, historical image data, such as fluoroscopy procedures, obtained from a large number of procedures. The historical image data may comprise: (i) static or moving devices (e.g., catheter, micro-catheter, guidewire, or any other device); (ii) static or moving background anatomy (including the planned path to the target site); and / or (ii) segmentation maps of devices and vessels per frame, if available. The training data may also comprise other information. At step 320 of the method, a training system trains the machine learning model, using the training data, to determine a minimum frame rate that maintains a required image quality. At step 330 of the method, the trained machine learning model 262 is stored for future use.

[0082] According to an embodiment, the patient procedure imaging system 200 is configured to process many thousands or millions of datapoints in the input data used to train the machine learning model 262, such as via the training instructions 263. For example, generating a functional and skilled trained machine learning model from a corpus of training data requires processing of millions of datapoints from input data and generated features. This can require millions or billions of calculations to generate a novel trained machine learning model from those millions ofdatapoints and millions or billions of calculations. As a result, each trained machine learning model is novel and distinct based on the input data and parameters of the machine learning algorithm, and thus improves the functioning of the system. Generating a functional and skilled trained machine learning model comprises a process with a volume of calculation and analysis that a human brain cannot accomplish in a lifetime, or multiple lifetimes.

[0083] According to an embodiment, reporting instructions 264 direct the system to provide the output of the system to a user, such as a clinician, via a user interface. The provided output can be any of the information as described or otherwise envisioned herein. The system may provide the information to a user via any mechanism, including but not limited to a visual display, an audible notification, a page, or any other method of notification. The information may be communicated by wired and / or wireless communication to another device. For example, the system may communicate the information to a mobile phone, computer, laptop, wearable device, and / or any other device configured to allow display and / or other communication of the information.

[0084] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0085] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”

[0086] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified.

[0087] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, whenused in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0088] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.

[0089] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are recited.

[0090] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively.

[0091] While several inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the inventive teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is,therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

Claims

ClaimsWhat is claimed is:

1. An imaging system (200) for performing a patient procedure, the system comprising: a processor (220) configured to: obtain one or more images acquired at an initial predetermined frame rate, analyze the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure, adjust the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate, obtain one or more new images acquired at the adjusted frame rate, predict one or more images when the determined minimum required frame rate is below a predetermined rate, and combine the one or more new images at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

2. The system of claim 1, further comprising a radiation source (270) configured to acquire images of a patient during the patient procedure.

3. The system of claim 1, wherein the processor is further configured to apply a trained machine learning model (262) configured to determine the minimum required frame rate for the subsequent window of the patient procedure.

4. The system of claim 3, wherein the machine learning model is trained to predict the one or more predicted images when the determined minimum required frame rate is below a predetermined rate.

5. The system of claim 1, further comprising a user interface (240) configured to provide the complete image sequence.

6. The system of claim 1, wherein the processor is further configured to obtain additional navigation data from a navigation data source, and wherein analyzing the one or more images to determine the minimum required frame rate for a subsequent window of the patient procedure further comprises analyzing the additional navigation data.

7. The system of claim 6, wherein the navigation data source is a second imaging modality (280).

8. The system of claim 6, wherein the additional navigation data is generated by the navigation source before the patient procedure.

9. The system of claim 6, wherein the additional navigation data is generated by the navigation source during at least a portion of the patient procedure.

10. The system of claim 1, further comprising: a catheter utilized inside the patient during the patient procedure; wherein the complete image sequence is utilized for navigation of the catheter.

11. The system of claim 3, wherein another processor is configured to train a machine learning model to generate the trained machine learning model (262), the another processor configured to: obtain (310) training data comprising imaging data from a plurality of patient procedures; train (320) the machine learning model to determine a minimum required frame rate for a subsequent window of a patient procedure, wherein the minimum required frame rate minimizes the frame rate while maintaining an image quality necessary to successfully perform the patient procedure; and store (330) the trained machine learning model in memory.

12. The system of claim 11, wherein the imaging data is obtained at a frame rate at or above the initial predetermined frame rate.

13. The system of claim 10, wherein, to train the machine learning model, the another processor is configured to identify, by the machine learning model, one or more factors within the training data influencing the minimum required frame rate.

14. The system of claim 13, wherein the one or more factors comprises one or more of the patient procedure being performed, an anatomy of the patient, and a behavior of a device being navigated inside the patient including translational velocity or rotational velocity of the device.

15. A method for performing a patient procedure, the method comprising: obtaining one or more images of a patient acquired at an initial predetermined frame rate; analyzing the one or more images to determine a minimum required frame rate for a subsequent window of the patient procedure; adjusting the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate; obtaining one or more new images acquired at the adjusted frame rate; predicting one or more images when the determined minimum required frame rate is below a predetermined rate; and combining the one or more new images at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

16. The method of claim 15, further comprising applying a trained machine learning model (262) configured to determine the minimum required frame rate for the subsequent window of the patient procedure and to predict the one or more predicted images when the determined minimum required frame rate is below a predetermined rate.

17. The method of claim 16, further comprising generating the trained machine learning model (262) by: obtaining (310) training data comprising imaging data from a plurality of patient procedures;training (320) the machine learning model to determine a minimum required frame rate for a subsequent window of a patient procedure, wherein the minimum required frame rate minimizes the frame rate while maintaining an image quality necessary to successfully perform the patient procedure; and storing (330) the trained machine learning model in memory.

18. The method of claim 17, wherein the training of the machine learning model comprises identifying, by the machine learning model, one or more factors within the training data influencing the minimum required frame rate.

19. A non-transitory computer-readable storage medium having stored a computer program comprising instructions, which, when executed by a processor, cause the processor to: obtain one or more images acquired at an initial predetermined frame rate; analyze the one or more images to determine a minimum required frame rate for a subsequent window of a patient procedure; adjust the initial predetermined frame rate to the determined minimum required frame rate, resulting in an adjusted frame rate; obtain one or more new images acquired at the adjusted frame rate; predict one or more images when the determined minimum required frame rate is below a predetermined rate; and combine the one or more new images at the adjusted frame rate and the predicted one or more images to generate a complete image sequence.

20. The non-transitory computer-readable storage medium, wherein the instructions, when executed by the processor, further cause the processor to to apply a trained machine learning model (262) configured to determine the minimum required frame rate for the subsequent window of the patient procedure.

Images