Dynamic visualization for device delivery

Abstract

The present disclosure provides systems and methods for dynamic visualization of delivery of a device within a blood vessel by correlating at least one first extraluminal image with a second extraluminal image. Extraluminal images may be correlated based on movement characteristics without the use of other sensors or timestamps. The first extraluminal image may be a high dose contrast agent X-ray angiography sheet ("XA"), and the second extraluminal image may be a low dose contrast agent XA. High dose contrast agents XA may be used to generate angiograms. The low dose contrast agent XA may be captured during delivery of a device (e.g., a balloon, stent, probe, or the like). Correlating the high dose XA and the low dose XA based on the movement features allows overlying the vasculature on the low dose XA to provide the physician with a visualization of the location of the device within the vessel tree in real time.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the filing dates of U.S. Provisional Patent Application No. 63 / 531,345 filed on August 8, 2023, U.S. Provisional Patent Application No. 63 / 603,931 filed on November 29, 2023, and U.S. Provisional Patent Application No. 63 / 659,536 filed on June 13, 2024, the disclosures of which are hereby incorporated by reference into this document. Background Art

[0003] Without the use of a radiocontrast agent (which may be more simply referred to as a "contrast agent" or just "contrast agent") (e.g., iodinated contrast agents) within a vessel, blood vessels are typically difficult to see on an X-ray angiogram. When an intravascular device (such as a balloon, stent, or probe) is delivered within a vessel, the physician typically refers to a vascular tree generated from an X-ray angiogram (XA) captured in the presence of high levels of radiocontrast agent to better define the vascular tree in the XA. However, the intravascular device itself is typically not visible in an X-ray angiogram captured with such high levels of contrast agent, and repeated or continuous use of contrast agent during a procedure is undesirable, at least for medical reasons. Typically, when an intravascular device is delivered, an X-ray angiogram may be captured with low levels of contrast agent or no contrast agent present in the vessel of interest so that the location of the device can be easily seen using the X-ray angiogram. From time to time, then, to better see the location of the device within the vascular tree, the physician may inject more contrast agent into the vessel of interest. The contrast agent is typically cleared after less than a second by cardiac function. To provide continuous visualization of the vascular tree during delivery of the device, the physician may either repeatedly inject more contrast agent, which may result in undesirable side effects, or fail to visualize the vascular tree while the device is being delivered. Summary of the invention

[0004] The present disclosure is generally directed to dynamically visualizing the delivery of a device within a vessel (e.g., in the context of a vessel tree or other region of interest) by correlating at least one first extraluminal image with a second extraluminal image. The extraluminal images can be correlated based on moving features visible within or derivable from the extraluminal images without the use of other sensors (e.g., readings from an electrocardiogram) or timestamps. The at least one first extraluminal image can be a high-dose contrast X-ray angiogram ("XA") captured using a high level of contrast agent present so that the vessel or vessel tree is clearly visible in the XA (referred to herein as high-dose contrast agent XA or high-dose XA). Then, the second extraluminal image can be a low-dose XA taken with a small amount of contrast agent or no contrast agent present in the vessel or vessel tree so that the intravascular device is more clearly visible (referred to herein as low-dose contrast agent XA or low-dose XA). Typically, for example, when the at least one high-dose XA is taken, the amount or concentration of contrast agent present in the vessel or vessel of interest can be at least five times or at least ten times the amount or concentration present when the low-dose XA is taken. In some examples, the second extraluminal image can be one or more subsequent images captured after capturing the first extraluminal image. In some examples, the second extraluminal image can take the form of a video or video stream of a low-dose contrast agent XA, which can be displayed in real time as a live low-dose XA. In some examples, the live low-dose XA can be a low-dose XA captured during an intravascular procedure (e.g., intravascular imaging, stent placement, etc.). In some examples, the live low-dose XA can be referred to as a fluoroscopic image.

[0005] At least one high-dose contrast agent XA may be used to generate a vascular map. A low-dose contrast agent XA may be captured during delivery of an intravascular device (e.g., a balloon, stent, probe, or the like). A moving feature may be used to achieve correlation between at least one high-dose XA and a low-dose XA or maintain a mapping between them, and the moving feature is typically a feature that is visible or detectable in both the high-dose XA and the low-dose XA or that can be derived from both the high-dose XA and the low-dose XA. For example, the moving feature may include at least one of a guide catheter tip, a distal endpoint of a working vessel, an optical flow at the guide catheter tip, or an optical flow at a distal endpoint of a working vessel.

[0006] The correlation or mapping then allows the vessel map to be overlaid on the low-dose XA, or vice versa, to provide the physician with a visualization of the location of the intravascular device within the vessel or vessel tree in real time. For example, this can allow the intravascular device to be tracked on the live low-dose XA with reference to the vessel map created using the at least one high-dose XA. In some examples, the location of the intravascular device can be determined based on the live low-dose XA, and an indication of the location can be provided on both the low-dose XA and the vessel map determined using the at least one high-dose XA. According to some examples, correlating or mapping between the at least one high-dose XA and the low-dose XA can allow a treatment zone (e.g., a stent or balloon landing zone) identified on the high-dose XA to be transferred or superimposed on the low-dose XA.

[0007] According to some examples, during a percutaneous coronary intervention (PCI) workflow, the at least one high-dose contrast agent XA can form at least a portion of a high-dose XA video, which can include both a pre-contrast low-dose XA captured before the contrast agent has been released into the blood vessel and a post-contrast high-dose XA captured once the contrast agent has been released. The post-contrast high-dose XA can provide a clear view of the vascular tree, including side branches that are usually not visible in the pre-contrast low-dose XA. The post-contrast high-dose XA can be used to generate a vascular map. The blood vessel of interest (e.g., a working vessel) can be identified based on the intravascular device (e.g., a guide catheter and a guide wire) in the pre-contrast low-dose XA. Then, the identified working vessel can be used to correlate or map between the low-dose XA and the high-dose XA captured during the delivery of the intravascular device.

[0008] This correlation or mapping provides for mapping a particular physical location captured in a low-dose XA (e.g., a location within a working vessel) to the same physical location in a high-dose XA or vessel map (e.g., the same location within a working vessel). In this manner, for example, features of the location of an intravascular device as seen or detected in a low-dose XA can be stably depicted within the context of one or more high-dose XAs or corresponding vessel maps, or features of features of a vessel map, for example, determined using one or more high-dose XAs can be stably depicted within the context of one or more low-dose XAs.

[0009] Working vessels can be identified using one or more artificial intelligence (AI) models (e.g., machine learning (ML) models) that use shape priors (e.g., predictions of the location of the wire and / or guide catheter in a previous frame) to predict the location of the wire and / or guide catheter in the current frame. The AI ​​model can be a spatiotemporal model. The AI ​​model can be trained based on annotated XA. In some examples, the annotations can be a set of lines that follow the trajectory of the guide catheter and wire in the pre-contrast XA.

[0010] Motion features (typically features visible or detectable in or derivable from both high-dose XA and low-dose XA) can be used to correlate or determine a mapping between high-dose XA and low-dose XA to provide dynamic visualization during delivery of a device within a vessel. Due to the continuous movement of the vessel during capture of the low-dose XA (e.g., due to cardiac motion if the vessel is located within cardiac tissue), the correlation or mapping may change continuously or rapidly (albeit in a repeating cycle caused by, for example, a heartbeat), and the motion features of the low-dose XA can be used to continuously update the correlation or mapping. Motion features can be identified in high-dose XA and low-dose XA. Motion features can be determined at the vessel level, the vessel region of interest level, and the vessel pixel level to characterize the motion phase of high-dose XA and low-dose XA. Motion features can be used to perform an initial search to correlate high-dose XA and low-dose XA with substantially the same temporal phase. Normalized cross-correlation can be performed on pairs of motion features in high-dose XA and low-dose XA. For a given pair of moving features, a sample of low-dose XA features within a predetermined time window can be identified. The time window can be compared to all windows of substantially the same time length from the corresponding high-dose XA features. A maximum correlation coefficient can be determined based on the more correlated high-dose moving feature and low-dose moving feature pairs. In some examples, an initial time shift can be determined. The initial time shift can be filtered using an online update. The online update can use a filter (e.g., a linear Kalman filter) to predict the time shift that is most likely to match the most closely acquired low-dose XA with the next available high-dose XA.

[0011] Based on the temporal correlation of the motion signatures, the high-dose XA and the low-dose XA may be correlated or mapped to each other without using any external sensor information. In some examples, in addition or as an alternative, the high-dose XA and the low-dose XA may be correlated using spatial landmarks.

[0012] According to some examples, the intravascular device may include one or more markers (e.g., radiopaque markers) visible on the low-dose XA. The markers may be detected and used to track the position of the intravascular device relative to the vascular map generated based on the high-dose XA. Tracking the device markers may allow the physician to determine whether the device is approaching and / or reaching the treatment zone identified on the vascular map. In some examples, based on the correlation of the low-dose XA and the high-dose XA, the treatment zone identified on the vessel may be reflected on the live low-dose XA. By transferring the annotations generated on the vascular map (e.g., the high-dose XA) to the live low-dose XA, the physician is able to dynamically visualize and track the position of the device relative to the device vascular morphology during the delivery of the device.

[0013] One aspect of the present technology generally relates to a computer-implemented method (e.g., a method implemented using one or more suitable computer processors) comprising: receiving at least one first extraluminal image; receiving a second extraluminal image captured during delivery of an intravascular device; detecting a motion feature in the at least one first extraluminal image and the second extraluminal image; correlating the at least one first extraluminal image and the second extraluminal image based on the detected motion feature; and providing, for output, a real-time visualization of a position of an intravascular device on one of the at least one first extraluminal image or the second extraluminal image that includes the intravascular device. The method may also include displaying the output of the real-time visualization on a suitable display device (e.g., a computer monitor).

[0014] The at least one first extraluminal image may be a high-dose contrast X-ray angiogram. The second extraluminal image may be a low-dose contrast X-ray angiogram. The intravascular device may be at least one of a stent delivery device, a balloon device, an intravascular imaging probe, a vessel preparation device, or a pressure line. The method may further include generating, by one or more processors, a vessel map based on the at least one first extraluminal image.

[0015] The method may further include automatically detecting the working vessel using the AI ​​model by the one or more processors. Training the AI ​​model to automatically detect the working vessel may include: annotating the pre-contrast extraluminal image as a line that follows the trajectory of at least one of the guide wire or the guide catheter, marking the path of the working vessel in the post-contrast extraluminal image, providing the annotated pre-contrast extraluminal image and the annotated post-contrast extraluminal image to the AI ​​model as training data, and training the AI ​​model to predict the working vessel trajectory. The method may further include: receiving at least one pre-contrast extraluminal image as input into the AI ​​model by the one or more processors; training the AI ​​model by the one or more processors that enhance the high-dose extraluminal image to a low-dose extraluminal image as input to segment at least one of the guide catheter, the guide wire, the stent marker, or the balloon marker on the low-dose extraluminal image; detecting the guide wire of the intravascular device based on the at least one pre-contrast extraluminal image by the one or more processors executing the AI ​​model; propagating cable information based on the detected guide wire by the one or more processors executing the AI ​​model on a frame-by-frame basis; and automatically predicting the working vessel trajectory by the one or more processors executing the AI ​​model based on the propagated cable information.

[0016] The movement feature may include at least one of the guide catheter tip, the distal end of the working vessel, the optical flow at the guide catheter tip, or the optical flow at the distal end of the working vessel. Detecting the movement feature may further include automatically detecting the working vessel by one or more processors executing the AI ​​model. Correlating the at least one first extraluminal image and the second extraluminal image may include: determining the vessel level movement by one or more processors, determining the cable tip level movement by one or more processors, and determining the vessel pixel level movement by one or more processors. Determining the vessel level movement may further include determining a two-dimensional translation vector by one or more processors. The two-dimensional translation vector may correspond to a two-dimensional translation relative to the first frame at the nth frame. Determining the cable tip level movement may include determining absolute spatial information and relative spatial information by one or more processors. The absolute spatial information may correspond to coordinates, and the relative spatial information corresponds to optical flow between adjacent image frames. Determining the vessel pixel level movement may include determining absolute spatial information and relative spatial information of multiple points on the working vessel by one or more processors. The absolute spatial information may correspond to coordinates, and the relative spatial information corresponds to optical flow between adjacent image frames. The method may further include determining the patient's heartbeat cycle based on the detected movement feature.

[0017] The method may further include determining, by one or more processors, a spatiotemporal phase match between at least one of the at least one first extraluminal image and the second extraluminal image based on the detected movement features. The method may further include resampling, by one or more processors, the detected movement features at a common frame rate, determining, by one or more processors, a maximum correlation coefficient for a pair of movement features in the at least one first extraluminal image and the second extraluminal image, and determining, by one or more processors, a time shift based on the maximum correlation coefficient. The method may further include determining, by one or more processors, a drift between movement features detected in the at least one first extraluminal image, and adjusting, by one or more processors, the time shift based on the determined drift. The method may further include iteratively predicting, by one or more processors, the time shift; and updating, by the one or more processors, the time shift to a corrected time shift based on the iteratively predicted time shift.

[0018] The method may further include adjusting the spatiotemporal phase matching by one or more processors. Adjusting the spatiotemporal phase matching may include identifying, by the one or more processors, another first extraluminal image that is different from at least one of the first extraluminal images, and adjusting the spatiotemporal phase matching by the one or more processors based on the another first extraluminal image. The method may further include updating, by the one or more processors, the real-time visualization to include the another first extraluminal image based on the adjustment.

[0019] The method may further include detecting, by one or more processors, an intravascular device. Detecting the intravascular device in the second extraluminal image may further include executing an AI model. The method may further include training the AI ​​model, wherein training the AI ​​model includes providing a co-registration dataset as input to the AI ​​model, the co-registration dataset including a plurality of intraluminal images and extraluminal images, wherein the plurality of intraluminal images and extraluminal images are annotated images; and training the AI ​​model to predict the location of the intravascular device. The annotated images include annotations identifying one or more intravascular device markers. The method may further include detecting, by one or more processors, an optical flow of the intravascular device; determining, by one or more processors, a location of the intravascular device in a first frame of the second extraluminal image based on the detected optical flow, and predicting, by one or more processors, a location of the intravascular device in subsequent frames of the second extraluminal image based on the detected optical flow.

[0020] The method may further include providing, by one or more processors, a treatment zone on at least one of the second extraluminal images or the at least one first extraluminal image for output. The treatment zone is at least one of a treatment device landing zone, a stent landing zone, a balloon device zone, a vessel preparation device zone, or a lesion-related zone. The lesion-related zone may be at least one of a calcification frame, a lipid frame, or an anatomical frame, and the lesion-related zone may be identified by one or more processors from another imaging modality and co-registered to the first extraluminal image. The other imaging modality may be an intravascular imaging modality.

[0021] The method may further include receiving, by one or more processors, annotations of the at least one first extraluminal image or the second extraluminal image. When receiving the annotations, the method may further include receiving, by one or more processors, one or more inputs from a user corresponding to the annotations, or automatically determining, by one or more processors, the annotations based on the vascular data. The annotations may include one or more of the following: plaque burden, fractional flow reserve ("FFR") measurements at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent decisions, scores, debulking recommendations, follow-up surgery recommendations, stent placement areas, treatment device landing areas, balloon device areas, vascular preparation device areas, or lesion-related areas. The method may further include updating, by one or more processors, the second of the at least one first extraluminal image or the second of the second extraluminal image based on the received annotations.

[0022] The method may further include automatically capturing, by one or more processors, a screenshot of a real-time visualization of the location of the intravascular device. The screenshot may be automatically captured when the intravascular device is within a threshold distance of an area of ​​interest. The area of ​​interest may be a treatment zone. The treatment zone may be at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone. Determining the threshold distance may include at least one of: determining, by one or more processors, the number of pixels between an outer boundary of the area of ​​interest and at least one detected marker on the intravascular device, or determining, by one or more processors, a spatial distance between at least one detected marker on the intravascular device and the area of ​​interest. The spatial distance may be a Euclidean distance, a geodesic distance, or the like.

[0023] The method may further include automatically zooming, by one or more processors, a portion of the real-time visualization. When the intravascular device is within a threshold distance of the region of interest, the automatic zoom may focus on the portion of the real-time visualization. The portion of the real-time visualization may correspond to the region of interest. The region of interest may be a treatment zone. The treatment zone may be at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone. Determining the threshold distance may include at least one of: determining, by one or more processors, the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or determining, by one or more processors, a spatial distance between at least one detected marker on the intravascular device and the region of interest. The spatial distance may be, for example, a Euclidean distance or a geodesic distance. In some examples, the automatic zoom may further include sorting, by one or more processors, pixel values ​​of the real-time visualization by their intensity, normalizing pixel intensity values ​​below a predetermined threshold, and applying a median filter to the normalized pixel intensity values ​​of the real-time visualization.

[0024] Another aspect of the present disclosure relates to a system including one or more processors. The one or more processors may be configured to receive at least one first extraluminal image, receive a second extraluminal image captured during delivery of an intravascular device, detect movement features in the at least one first extraluminal image and the second extraluminal image, correlate the at least one first extraluminal image and the second extraluminal image based on the detected movement features, and provide a real-time visualization of a position of the intravascular device on one of the at least one first extraluminal image or the second extraluminal image including the intravascular device for output.

[0025] Yet another aspect of the present disclosure relates to a non-transitory computer-readable medium which, when executed by one or more processors, causes the one or more processors to receive at least one first extraluminal image, receive a second extraluminal image captured during delivery of an intravascular device, detect movement in the at least one first extraluminal image and the second extraluminal image, correlate the at least one first extraluminal image and the second extraluminal image based on the detected movement characteristics, and provide real-time visualization of a position of the intravascular device on one of the at least one first extraluminal image or the second extraluminal image including the intravascular device for output.

[0026] One aspect of the technology relates to a method comprising receiving an extraluminal image captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detecting a plurality of device marker candidates corresponding to the radiopaque marker of the intravascular device; and automatically detecting a working vessel using a plurality of virtual boxes and at least one of the plurality of device marker candidates using an artificial intelligence (AI) model. At least one of the plurality of virtual boxes contains at least one of the plurality of device marker candidates. The method further comprises selecting, by one or more processors, at least one of the plurality of virtual boxes including a region of interest of the working vessel, the region of interest containing at least one of the plurality of device markers.

[0027] In some examples, the method may further include predicting a center point of each of the plurality of virtual boxes, pairing the detected device marker with the closest device marker in the plurality of virtual boxes, filtering device marker candidates beyond the boundaries of the plurality of virtual boxes, updating the center point of each of the plurality of virtual boxes using the filtered device marker candidate points, determining a displacement of the predicted center point and the updated center point, and repositioning the plurality of virtual boxes based on the determined displacement. Updating the center point may include approximating the center point of the box using the device marker candidate. One or more device marker candidates may be located within at least one of the plurality of boxes.

[0028] In some examples, the AI ​​model can be trained using annotations on high-dose angiograms. In some examples, the extraluminal image can be a live X-ray angiogram or a fluoroscopic image.

[0029] In some examples, the method may further include tracking a region of interest during a percutaneous coronary intervention procedure. The method may further include enhancing the region of interest using local contrast stretching by selectively enhancing the local contrast between the detected device marker candidate and the surrounding area. In some examples, the method may further include automatically scaling the region of interest by one or more processors. Automatic scaling may include sorting, by one or more processors, pixel values ​​for real-time visualization by their intensity, normalizing pixel intensity values ​​below a predetermined threshold, and applying a median filter to the normalized pixel intensity values ​​for real-time visualization.

[0030] Another aspect of the present disclosure relates to a system comprising one or more processors configured to receive an extraluminal image captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detect a plurality of device marker candidates, wherein at least one of the device marker candidates corresponds to the radiopaque marker of the intravascular device; automatically detect a working vessel using a plurality of virtual boxes and at least one of a plurality of device marker candidates using an artificial intelligence model, wherein at least one of the plurality of virtual boxes contains at least one of the plurality of device marker candidates; and select at least one of the plurality of virtual boxes including an area of ​​interest of the working vessel, the area of ​​interest containing at least one of the plurality of device markers.

[0031] Yet another aspect of the present disclosure relates to a non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause: the one or more processors to receive an extraluminal image captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detect a plurality of device marker candidates, wherein at least one of the device marker candidates corresponds to the radiopaque marker of the intravascular device; use an artificial intelligence model to automatically detect a working vessel using a plurality of virtual boxes and at least one of a plurality of device marker candidates, wherein at least one of the plurality of device marker candidates contains at least one of the plurality of device marker candidates; and select at least one of a plurality of virtual boxes including an area of ​​interest of the working vessel, the area of ​​interest containing at least one of the plurality of device markers. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is an exemplary data collection system for collecting intravascular data and extravascular data according to aspects of the present disclosure.

[0033] Figure 2 is a diagram according to various aspects of the present disclosure, for example Figure 1 Flow chart of a method for dynamic visualization of an intravascular device within a system.

[0034] Figure 3A is a flow chart illustrating a method for identifying working vessels and device markers according to various aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0035] Figure 3B According to various aspects of the present disclosure, Figure 3A An exemplary skeleton frame used in the method.

[0036] Figure 3C is a diagram according to various aspects of the present disclosure for use in Figure 3AA flow chart of the alignment method is shown with markers used in the method.

[0037] Figure 4 is a flowchart of a method for generating shape priors according to various aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0038] Figure 5A is a spatiotemporal encoder-decoder segmentation network according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0039] Figure 5B A method for training a Figure 5A Annotations of the AI ​​models used in the network.

[0040] Figure 6 is a flow chart illustrating a method of mobile phase matching according to various aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0041] Fig. 7A is a graph for determining movement characteristics according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0042] Figure 7B According to various aspects of the present disclosure, Fig. 7A A graphical representation of the periodic pattern of the mobile features described in the figure.

[0043] Figure 8 is a flowchart illustrating a method of temporal phase matching according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0044] Fig. 9 is a flow chart illustrating a method of initial search for a temporal phase match according to aspects of the present disclosure, which may be performed in Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0045] Fig.10 is a flowchart illustrating a method of updating an initial search for temporal phase matching according to aspects of the present disclosure, which may be performed in Figure 1data collection system to achieve Figure 2 Dynamic visualization method.

[0046] Fig.11 is a flowchart illustrating a method for determining a heartbeat cycle according to various aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0047] Fig.12 is a flow chart illustrating a method of detecting and tracking device tags according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0048] Fig.13 is a flowchart illustrating a method of determining a marker position according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0049] Fig.14 is an example of marker detection and tracking according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0050] Fig.15 is an example of a landing zone that can be identified according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0051] Fig.16 is a flow chart illustrating a method for dynamically visualizing the delivery of a device according to aspects of the present disclosure, which may be performed in Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0052] Fig.17 is a flow chart illustrating a method for dynamically visualizing the delivery of a device according to aspects of the present disclosure, which may be performed in Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0053] Fig.18 is an example of adjusting the time synchronization of the extraluminal images according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 Dynamic visualization method.

[0054] Fig.19 is another example of adjusting the time synchronization of the extraluminal images according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0055] Fig. 20 is another example of adjusting the time synchronization of the extraluminal images according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0056] Fig.21 is an example of real-time state transition for dynamic visualization of the delivery of a device according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0057] Fig. 22 Figure 3 illustrates the determination of device marker candidates and shape priors from extraluminal images, which can be used in Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0058] Fig.23 is an exemplary annotation for training an artificial intelligence model according to aspects of the present disclosure, which may be found at Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0059] Fig.24 is an illustration of annotations overlaid on an XA image according to aspects of the present disclosure, which may be viewed in Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0060] Fig.25 is an illustration of a method of defining a region of interest and propagating a region of interest according to aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0061] Fig.26 is a flowchart illustrating a method for determining a region of interest for a device marker according to various aspects of the present disclosure, which may be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0062] Fig.27A is a real-time local contrast stretching for enhancing the appearance of device markers in extraluminal images according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0063] Fig.27B is another exemplary real-time local contrast stretching for enhancing the appearance of device markers in extraluminal images according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method.

[0064] Fig.27C is another exemplary real-time local contrast stretching for enhancing the appearance of device markers in extraluminal images according to aspects of the present disclosure, which can be Figure 1 data collection system to achieve Figure 2 A dynamic visualization method. DETAILED DESCRIPTION

[0065] The technology generally involves dynamic visualization of an intravascular device within a blood vessel by correlating or mapping between extraluminal images. The extraluminal images can be correlated based on visible or detectable moving features within the images themselves, without the use of other sensors (e.g., readings from an electrocardiogram), and without the use of timestamps. The extraluminal images can include a pre-contrast X-ray angiogram ("XA"), a high-dose post-contrast XA, and a subsequent low-dose XA. For example, the high-dose XA can be used to generate a vessel map, the pre-contrast XA can be used to identify a vessel of interest or a working vessel, and the low-dose XA can be taken during delivery of the device and used to track the position of the intravascular device.

[0066] Compared with low-dose contrast agent XA images, high-dose contrast agent XA images use more contrast agent during imaging. According to some examples, high-dose XA can use 8-10cc of contrast agent, while low-dose XA can use 2-3cc of contrast agent. In some examples, high-dose XA can use at least three to five times more contrast agent than low-dose XA. In another example, high-dose XA can use at least five times or more contrast agent than low-dose XA. High-dose XA images can expose patients to higher radiation doses than low-dose contrast agent XA images. Compared with low-dose contrast agent XA images, high-dose XA images can have longer imaging times. Compared with low-dose XA images with lower contrast agent and / or lower radiation dose, high-dose XA images can have better diagnostic quality due to the increase of contrast agent and / or higher radiation dose. As used herein, high-dose contrast agent XA can be interchangeable with high-dose XA, and low-dose contrast agent XA can be interchangeable with low-dose XA.

[0067] According to some examples, the wire and / or guide catheter used to deliver the device can be used to identify the working vessel. The wire and guide catheter can be visible in the pre-contrast XA. For example, the system can integrate artificial intelligence ("AI") to identify the working vessel. As an example, the AI ​​can be an AI ("ML") model trained to segment the XA to predict an output category, which includes a guide catheter, a wire, a catheter marker, and a background. The multi-annotated segmentation output by the ML model can be converted into a binary mask that illustrates the working vessel region as a shape prior for subsequent XA. According to some examples, the initial shape prior can be obtained from the pre-contrast XA. The initial shape prior determined based on the pre-contrast XA can be used to segment the working vessel in the subsequent high-dose XA. According to some examples, the AI ​​may include a second ML model that is trained to identify the working vessel based on the shape prior determined by the first ML model. The second ML model can be a spatiotemporal model. The second ML model can receive as input the prediction information of the current XA frame plus the previous XA frame. For example, the prediction information can be a prediction of the position of the working vessel in a given XA frame. In some examples, the second ML model may receive a shape prior of a previous frame as an additional input. The second ML model may provide an identified working vessel as an output. According to some examples, the AI ​​may be used to segment the guide catheter and the cable tip so that the locations of the endpoints of the guide catheter and the wire tip can be automatically detected based on the segmentation results. The AI ​​may include other ML models similar to the first ML model and / or the second ML model.

[0068] The motion feature library can be generated based on the motion features detected in the high-dose XA and the low-dose XA. The feature library can provide various aspects of motion information and can be used to phase match the low-dose XA and the high-dose XA of a given patient. The motion features can be used to perform initial phase matching of the low-dose XA and the high-dose XA. The motion features can include the guide catheter tip, the distal end point of the working vessel, the optical flow at the guide catheter tip, and / or the optical flow at the distal end point of the working vessel. The initial phase match can be an initial time phase match. The high-dose XA and the low-dose XA can be matched in time without using EKG, or other physiological signals that measure the patient's heartbeat or breathing pattern, or timestamps. Instead, the time phase matching can be based on image motion features.

[0069] According to some examples, an update may be performed to iteratively filter an initial time shift estimate. The initial time shift estimate may be updated during an online update. For example, a filter (such as a linear Kalman filter) may be used to predict a time shift based on a more current low dose XA and a next available high dose XA. The predicted time shift may be used to cross-correlate the low dose XA and the high dose XA to determine whether the filter should be updated. The time shift determined by the online update may be used to correlate the low dose XA and the high dose XA. In some examples, the spatial adaptability between spatial landmarks (e.g., guide catheters, cable tips, devices, or the like) identified in the high dose XA and the low dose XA may be used to evaluate and / or refine the phase matching performed using the moving feature.

[0070] According to some examples, the motion feature can be used to derive the patient's heartbeat cycle. For example, an autocorrelation function can be applied to the motion feature. The heartbeat cycle can be detected based on voting and non-maximum suppression.

[0071] In order to provide visualization of the device during the delivery and / or treatment phase, markers on the device can be detected and tracked. The device may include one or more radiopaque markers that can be detected using high-dose XA and / or low-dose XA. The device can be detected and tracked by iteratively estimating the position of the markers in the XA. In some examples, the movement of the markers within a sampling period is estimated by optical flow methods. The movement estimate can be used to predict the position of the marker in the latest XA.

[0072] Correlating live low-dose XA with high-dose XA can reduce the amount of contrast required during PCI. For example, by identifying motion features in both low-dose XA and high-dose XA, low-dose XA can be correlated temporally and spatially so that the position of the device can be tracked in real time in low-dose XA relative to the vessel map generated from high-dose XA.

[0073] In some examples, using moving features to correlate live low-dose XA with high-dose XA can reduce the number of sensors and / or equipment required for co-registration of low-dose XA and high-dose XA. For example, by using moving features that can be detected in low-dose XA and high-dose XA, it is no longer necessary to synchronize the collection of images, timestamps, or the like. In addition, the use of EKG to co-register images based on the heartbeat cycle is negated. According to some examples, the amount of contrast agent required during surgery can be reduced by correlating low-dose XA and high-dose XA using moving features. For example, a doctor will be able to dynamically visualize the position of the device in the blood vessel based on the correlated low-dose XA and high-dose XA, without having to inject additional contrast agents to determine the position of the device in the blood vessel. On the contrary, dynamic visualization allows the doctor to see the position of the device on the blood vessel map in real time without using additional contrast agents.

[0074] According to some examples, by using AI to correlate low-dose XA and high-dose XA to provide dynamic visualization of delivery of a device within a blood vessel, the amount of computing resources required to generate the dynamic visualization is reduced. For example, the AI ​​may include one or more AI models. Using one or more AI models to correlate high-dose XA and low-dose XA can reduce the amount of memory and processing power required to dynamically visualize the device as it is delivered within a blood vessel.

[0075] Exemplary Systems

[0076] Figure 1 A data collection system 100 for collecting intravascular data as well as extravascular data is illustrated. System 100 may include a non-invasive imaging system 120, an intravascular device 104, a storage device 106, a subsystem 108, and a network 102. Subsystem 108 may include an optical receiver 110, a computing device 112, and a display 118.

[0077] The non-invasive imaging system 120 can be, for example, a nuclear magnetic resonance, an X-ray, a computer-assisted tomography, or other suitable non-invasive imaging technology. In particular, the non-invasive imaging system can be an angiography system that is configured to generate a pre-contrast X-ray angiogram (XA) before a radiopaque contrast agent has been used, to shoot a high-dose contrast agent XA when a radiopaque contrast agent is present in one or more blood vessels of interest, and to shoot a low-dose contrast agent XA when little or no contrast agent is present. In some examples, the high-dose contrast agent XA can be a movie image, and the low-dose contrast agent XA can be a fluoroscopic image. For example, the angiography system 120 can include a fluoroscopic system. The angiography system 120 is configured to non-invasively image the subject S so that frames of angiography data (usually in the form of frames of image data) are generated. According to some examples, X-ray imaging can occur while being delivered via the vascular delivery device 104, so that blood vessels in the region R of the subject S are imaged using angiography. In some examples, a high dose XA may be generated before, during, and / or after the intravascular device 104 is delivered via a blood vessel. Imaging results of the non-invasive scan may be provided for output on a display 118. The high dose XA and the low dose XA may be correlated so that the device and / or treatment area may be dynamically visualized during surgery. As discussed above and herein, the non-invasive imaging system may be an angiography system. However, the non-invasive imaging system 120 may use a variety of other imaging techniques, and therefore, reference to the non-invasive imaging system 120 being an angiography system is not intended to be limiting.

[0078] The non-invasive imaging system 120 can communicate with the storage device 106 via the network 102. In some examples, the imaging system 120 can communicate directly with the storage device 106 (e.g., without having to be connected via the network 102). The storage device 106 can be a workstation or a server. The data collected and / or generated by the imaging system 120 can be stored and managed by the storage system 106. In some examples, a subsystem, server, or workstation handles the functions of the storage device 106. According to some examples, the imaging system 120 can generate electromagnetic radiation (e.g., X-rays). In some examples, the imaging system 120 can receive such radiation after the radiation passes through the subject S. In turn, the data storage 106 can use the signal from the imaging system 120 to image one or more regions of the subject S including the region R. In some examples, the storage device 106 can be integrated with the imaging device 120.

[0079] The region of interest R may be a subset of the vascular system or the peripheral vascular system (such as a specific blood vessel). The region R may be an area for the delivery of the device 104. The device may be, for example, a stent, a balloon, or the like. In some examples, the device 104 may be an intravascular device (e.g., an optical coherence tomography ("OCT") probe, an intravascular ultrasound ("IVUS") catheter, a micro-OCT probe, a near-infrared spectroscopy (NIRS) sensor, optical frequency domain imaging ("OFDI"), or any other device that can be used to image a blood vessel. In some examples, the device 104 may be a pressure line, a flow meter, or the like. The device 104 may include a device tip, one or more radiation-impermeable markers, an optical fiber, a torque line, or the like. In addition, the device tip may also include one or more data collection subsystems, such as an optical beam guide, an acoustic beam guide, a pressure detector sensor, other sensors or detectors, and combinations of the foregoing components.

[0080] In examples where the device 104 includes a beam guide, the optical fiber can be in optical communication with the device 104 and / or with the beam guide. The torque line can define a hole in which the optical fiber is disposed. According to some examples, the device 104 can also include a sheath, such as a polymer sheath (not shown) that forms part of a catheter. The optical fiber, which is part of a sampling arm of an interferometer in the context of an intravascular system, can be optically coupled to the subsystem 108 and / or a patient interface unit (PIU).

[0081] A guide wire (not shown) may be used to introduce device 104 into the blood vessel. In examples where device 104 is an intravascular data collection device, device 104 may be introduced and withdrawn along the length of the blood vessel while collecting data. In examples where device 104 is a treatment device (e.g., a device for delivering a balloon or stent), device 104 may be introduced into the blood vessel and guided to the treatment area.

[0082] The device 104 may be connected to the subsystem 108. According to some examples, the device 104 may be connected to the subsystem 108 via an optical fiber. The subsystem 108 may include a light source (such as a laser), an interferometer with a sampling arm and a reference arm, various optical paths, a clock generator, a photodiode, and other OCT, IVUS, micro-OCT, NIRS, and / or pressure line components. The device 104 may be connected to an optical receiver 110. According to some examples, the optical receiver 110 may be a balanced photodiode-based system. The optical receiver 110 may be configured to receive light collected by the probe 104. The device 104 may be connected to the optical receiver 110 via a wired connection or a wireless connection.

[0083] The system 100 may further include or be configured to receive data from a non-invasive imaging system 120. For example, the non-invasive imaging system 120 may be an imaging system based on angiography, fluoroscopy, X-ray, magnetic resonance imaging, computer-assisted tomography, etc. The non-invasive imaging system 120 may be configured to non-invasively image the blood vessel 102. According to some examples, the non-invasive imaging system 120 may acquire one or more images before, during, and / or after the withdrawal of the data collection probe 104. The non-invasive imaging system 120 may be used to image the patient so that a diagnostic decision can be made and various possible treatment options (e.g., stent placement) can be performed. These and other imaging systems can be used to image the patient externally or internally to obtain raw data, which may include various types of image data.

[0084] The non-invasive imaging system 120 may communicate with the subsystem 108. According to some examples, the non-invasive imaging system 120 may be wirelessly coupled to the subsystem 108 via the network 102. For example, the non-invasive imaging system 120 may be wirelessly coupled to the subsystem 108 via a communication interface (e.g., Wi-Fi or Bluetooth). In some examples, the non-invasive imaging system 120 may communicate with the subsystem 108 via a cable (e.g., an optical fiber). In yet another example, the external imaging device 120 may be indirectly communicatively coupled to the subsystem 108 or the computing device 112. For example, the non-invasive imaging system 120 may be coupled to a separate computing device (not shown) that communicates with the computing device 112. As another example, data from the imaging system 120 may be transferred from the storage device 106 to the computing device 112 using a computer-readable storage medium, via the network 102, or the like.

[0085] The subsystem 108 may include a computing device 112. One or more steps may be performed automatically or without user input to navigate images, enter information, select and / or interact with input, etc. In some examples, one or more steps may be performed based on user input received via mouse clicks, keyboard, touch screen, verbal commands, etc. The computing device 112 may include one or more processors 113, memory 114, instructions 115, data 116, and one or more modules 117.

[0086] The one or more processors 113 may be any conventional processor, such as a commercially available microprocessor. Alternatively, the one or more processors may be a dedicated device, such as an application specific integrated circuit (ASIC), or other hardware-based processor. Figure 1The processor, memory, and other elements of device 112 are functionally illustrated as being within the same block, but one of ordinary skill in the art will appreciate that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories, which may or may not be stored within the same physical housing. Likewise, the memory may be a hard disk or other storage medium located in a housing different from the housing of device 112. Thus, references to a processor or computing device will be understood to include references to a collection of processors, computing devices, or memories that may or may not operate in parallel.

[0087] The memory 114 can store information accessible by the processor, including instructions 115 executable by the processor 113, and data 116. The memory 114 can be a type of memory operable to store information accessible by the processor 113, including non-transitory computer-readable media, or other media that stores data that can be read by electronic devices, such as hard disks, memory cards, read-only memories ("ROM"), random access memories ("RAM"), optical disks, and other writable and read-only memories. The subject matter disclosed herein can include different combinations of the above, whereby different portions of the instructions 115 and data 116 are stored on different types of media.

[0088] The memory 114 can be retrieved, stored or modified by the processor 113 according to the instructions 115. For example, although the present disclosure is not limited to a specific data structure, the data 116 can be stored in a computer register, stored in a relational database as a table with multiple different fields and records, stored in an XML document or a flat file. The data 116 can also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII codes or Unicode. By way of further example only, the data 116 can be stored as a bitmap composed of pixels (the bitmap is stored in a compressed or uncompressed manner), or various image formats (e.g., JPEG), vector-based formats (e.g., SVG), or computer instructions for drawing graphics. In addition, the data 116 can include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations), or information used by functions to calculate related data.

[0089] Instructions 115 may be any set of instructions (e.g., machine code) that are directly executed by processor 113, or any set of instructions (e.g., scripts) that are indirectly executed. In this regard, the terms "instructions," "applications," "steps," and "programs" may be used interchangeably herein. Instructions may be stored in object code format for direct processing by a processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The functions, methods, and routines of the instructions are explained in detail below.

[0090] According to some examples, the computing device 112 may include receiving data from the device 104 via a wired connection or via a wireless connection. The data may include, for example, intravascular data, including intravascular imaging data, pressure data, temperature data, flow data, or the like. The data received by the computing device 112 from the imaging system 120, the storage device 106, and / or the device 104 may be used to dynamically visualize the device 104 on one or more extraluminal images. In some examples where the device is an intravascular data collection device, the data received from the device may be used to determine plaque burden, fractional flow reserve ("FFR") measurements at one or more locations along the vessel, calcium angles, EEL detection, calcium detection, proximal frames, distal frames, EEL-based indicators, stent / no-stent decisions, scoring, recommendations for debulking and other procedures, evidence-based recommendations obtained by automatically detecting areas / features of interest, stent planning, and the like.

[0091] In an example, device 104 is an intravascular data collection device, and data obtained by device 104 and / or imaging system 120 can be processed by one or more modules to provide information about the blood vessel (including lumen contour, vessel diameter, vessel cross-sectional area, lumen area, EEL value, EEL diameter, EEL arc, lesion location, lesion size, plaque burden, VFR, FFR, landing zone, treatment zone, virtual stent defined by the landing zone, or the like). According to some examples, module 117 may include a dynamic visualization module, an EEL detection module, a lumen detection module, a lesion detection module, and a co-registration module.

[0092] The dynamic visualization module can automatically correlate the high dose XA and the low dose XA so that the vessel map can be overlaid on the live low dose XA, the treatment area identified on the high dose XA and / or the vessel map can be overlaid on the live low dose XA, and / or the device 104 can be tracked on the live low dose XA with reference to the vessel map. The dynamic visualization module can correlate the high dose XA and the low dose XA based on movement features detected in both the high dose XA and the low dose XA. In some examples, the dynamic visualization module can use AI (e.g., one or more AI ("ML") models) to identify working vessels, detect markers on the device, and track the movement of the device.

[0093] The EEL detection module may automatically detect and measure the EEL diameter of a given intravascular image frame taken during the pullback of the vessel. The lumen detection module may automatically detect and measure the lumen diameter of a given intravascular image frame taken during the pullback of the vessel. The lesion detection module may automatically detect lesions within the vessel based on the vessel data obtained from the device 104 and / or the imaging system 120. The side branch detection module may process the vessel data obtained from the device 104 and / or the imaging system 120 to detect one or more side branches of the vessel.

[0094] The computing device 112 may be adapted to co-register the vascular data obtained during the withdrawal of the device 104 with the intravascular image and / or the extraluminal image. For example, the computing device 112 may be configured to receive and store extraluminal image data, such as image data generated by the imaging system 120 and obtained by the frame grabber. The computing device 112 may be configured to receive and store intravascular image data, such as image data generated by the device 104 and obtained by the frame grabber. In some examples, the computing device 112 may access a co-registration module to co-register the vascular data with the luminal image. The luminal image may be an extraluminal image, such as an angiogram, an X-ray, or the like. The co-registration module may co-register the intravascular data (e.g., an intravascular image, plaque burden, EEL measurements, luminal diameter measurements, pressure readings, virtual flow reserve ("VFR"), fractional flow reserve ("FFR"), resting full cycle ratio ("RFR"), flow velocity, etc.) with the extraluminal image. In some examples, the co-registration module may co-register the intravascular data with an intraluminal image (eg, an intraluminal image captured by an OCT probe, an IVUS probe, a micro-OCT probe, or the like).

[0095] In one example, the co-registration module may co-register the intraluminal data captured during the pullback with one or more extraluminal images. For example, the extraluminal image frame may be pre-processed. For example, convolution matrices, Hessian matrices, and other various matrices may be applied on a per-pixel basis to change the intensity, remove, or otherwise modify a given angiographic image frame. As described herein, the pre-processing stage may enhance, modify, and / or remove features of the extraluminal image to increase the accuracy, processing speed, success rate, and other characteristics of subsequent processing stages. The vessel centerline may be determined and / or calculated. In some examples, the vessel centerline may be superimposed or otherwise displayed relative to the pre-processed extraluminal image. According to some examples, the vessel centerline may represent the trajectory of the device 104 (e.g., an intravascular device) through the vessel during the pullback. In some examples, the centerline may be referred to as a trace. Additionally or alternatively, a marker band or a radiopaque marker may also be detected in the extraluminal image frame. According to some examples, the extraluminal image frame and the data received by the device 104 may be co-registered based on the position of the determined marker band.

[0096] According to some examples, the modules may additionally or alternatively include a video processing software module, a preprocessing software module, an image file size reduction software module, a catheter removal software module, a shadow removal software module, a vessel enhancement software module, a speckle enhancement software module, a Gaussian Laplacian filter or transform software module, a wire detection software module, an anatomical feature detection software module, a fixed marker detection software module, a background subtraction module, a Frangi vessel software module, an image intensity sampling module, a moving marker software detection module, an iterative centerline test software module, a background subtraction software module, a morphological closure operation software module, a feature tracking software module, a catheter detection software module, a bottom hat filter software module, a path detection software module, a Dijkstra software module, a Viterbi software module, a fast marching method-based software module, a vessel centerline generation software module, a vessel centerline tracking module software module, a Hessian software module, an intensity sampling software module, an image intensity overlay software module, and other suitable software modules described herein. According to some examples, modules may include software (e.g., pre-processing software), transforms, matrices, and other software-based components that process image data or respond to patient triggers to facilitate co-registration of different types of image data by other software-based components, or otherwise perform annotations on image data to generate ground truth and other software, modules, and functions suitable for implementing various features of the present disclosure. Modules may include lumen detection using scanline-based or image-based methods, stent detection using scanline-based or image-based methods, indicator generation, generation of apposition bars for stent planning, wire shadow indicators to prevent confusion with bifurcation, side branches, and missing data, and others.

[0097] In some examples, the module may be configured to process vascular data obtained by device 104 and / or imaging system 120 using an AI algorithm, artificial intelligence, or the like.

[0098] The subsystem 108 may include a display 118 for outputting content to a user. The display 118 may be integrated with the computing device 112 or may be a separate unit electrically coupled to the computing device 112. The display 118 may output intravascular data related to one or more features detected in the vessel and / or obtained during the retrieval. For example, the output may include, but is not limited to, cross-sectional scan data, longitudinal scans, three-dimensional representations generated based on intraluminal images and / or extraluminal images, diameter maps, image masks, lumen boundaries, plaque size, plaque perimeter, visual indications of plaque location, visual indications of risk posed to stent expansion, flow rates, suggested treatment zones, or the like. The display 118 may identify features with text, arrows, color coding, highlights, outlines, or other suitable human or machine readable indicia.

[0099] According to some examples, display 118 may include a graphical user interface (“GUI”). Display 118 may be a touch screen display in which a user may provide input to navigate images, enter information, select input, and / or interact with input, etc. In some examples, display 118 and / or computing device 112 may include an input device, such as a trackpad, mouse, keyboard, etc., which allows a user to navigate images, enter information, select input, and / or interact with input, etc.

[0100] In some examples, the information input by the user can be an annotation. For example, the system can be configured to receive annotations to one or more representations of a vessel. In some examples, the annotation can be an indication of: plaque burden, fractional flow reserve ("FFR") measurements at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent / stentless decisions, scoring, recommendations for debulking and other surgeries, evidence-based recommendations obtained by automatically detecting areas / features of interest, stent planning, etc. In some examples, the annotation can be a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone. For example, the system can receive input corresponding to a proximal position and a distal position along a vessel, the proximal position and the distal position corresponding to a proximal position and a distal position of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

[0101] According to some examples, the annotations can be automatically determined by the system. For example, the system can determine one or more of the following based on the vascular data: plaque burden, FFR measurements at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent / stentless decision, scoring, recommendations for debulking and other procedures, evidence-based recommendations obtained by automatically detecting regions / features of interest, stent planning, treatment device landing zone, balloon device landing zone, vessel preparation device zone, lesion-related zone, etc. The system can automatically provide plaque burden, FFR measurements at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent / stentless decision, scoring, recommendations for debulking and other procedures, evidence-based recommendations obtained by automatically detecting regions / features of interest, stent planning, treatment device landing zone, balloon device landing zone, vessel preparation device zone, lesion-related zone, etc. as one or more annotation outputs on at least one of the vascular representations.

[0102] In some examples, input corresponding to input information and / or annotations (collectively referred to as "annotations") received by the system may be provided for output on the display 118. Annotations received about one vessel representation may be provided for display on the vessel representation where the annotation was received. In some examples, annotations may be provided for display on multiple vessel representations. For example, if the system receives an annotation about a high-dose XA, the annotation may be provided for output on one or both of the high-dose XA and the low-dose XA. In examples where there are additional representations (e.g., longitudinal representations, three-dimensional representations, or the like), annotations may be provided for output on one, some, or all representations. According to some examples, when the system receives an updated annotation on at least one of the representations, the annotations on the other representations may also be updated to correspond to the updated annotation.

[0103] The display screen 118, alone or in combination with the computing device 112, may allow switching between one or more viewing modes in response to user input. For example, a user may switch between different intravascular data, images, etc. recorded during each pullback. In some examples, the user may be able to switch between different representations (e.g., longitudinal representations, cross-sectional representations, three-dimensional representations, intravascular images, color images, black and white images, live images, or the like).

[0104] In some examples, the display 118, alone or in combination with the computing device 112, may present one or more menus as output to the physician, in response to which the physician may provide input by selecting an item from the one or more menus. For example, the menu may allow the user to display or hide various features. As another example, there may be a menu for selecting vascular features for display.

[0105] The output may include one or more representations of the blood vessel. For example, the representation may include image data, a longitudinal representation, a three-dimensional representation, a live representation, or the like. The image data may include, for example, a pre-contrast XA image, a high-dose contrast XA image, and a low-dose contrast XA image obtained and / or generated by the imaging system 120. In some examples, the output may include a cross-sectional image obtained by the device 104 during the withdrawal. In some examples, the output extraluminal image may include a blood vessel map generated based on a high-dose XA and a live low-dose XA image obtained during the delivery of the device. The low-dose XA may include an indication of the position of the device 104, an indication of the treatment zone initially identified on the high-dose XA, or the like. In some examples, the output may include a longitudinal representation of the blood vessel, a graphical representation of the blood vessel data, a three-dimensional representation of the blood vessel and / or blood vessel data, or the like. The representation of the blood vessel may provide views of various perspectives and cross-sections. In some examples, the EEL position, its diameter, or other EEL-based parameters may be output to display relative to the measured value of the angle, the detected calcium arc, the plaque load, or the like.

[0106] In some examples, one or more visual representations of the image may include an indication of: lesion location, lesion severity, lesion length, or the like. Additionally or alternatively, the indication of the lesion may be color coded, where each color represents a severity, length, or other measurement associated with the lesion.

[0107] In some examples, the output may include a candidate treatment zone. The candidate treatment zone may be, for example, a candidate stent landing zone. For example, the output may include an indication corresponding to a candidate proximal landing zone for the stent and a candidate distal landing zone for the stent. The candidate treatment zone may be determined based on a determined plaque load, lesion location, lesion length, or the like. The indication may be provided on any of a vessel representation (e.g., a three-dimensional representation, a longitudinal representation, a graphical representation, image data (e.g., an external image), etc.).

[0108] According to some examples, display 118 and / or computing device 112 may be configured to receive one or more inputs corresponding to selections of one or more representations. For example, input corresponding to selection of an image frame on a longitudinal representation may be received. In response, other representations provided for output may be updated to display corresponding indications or image frames. For example, an extraluminal image may be updated to have indications along a vessel corresponding to the location of the image frame selected in the longitudinal representation, a circumferential indication may be provided on a three-dimensional representation corresponding to the location of the image frame selected in the longitudinal representation, a cross-sectional image frame may be updated to correspond to the image frame selected in the longitudinal representation, and so on. In some examples, vascular data associated with the selected location may be updated and provided for display.

[0109] Dynamic Visualization

[0110] Figure 2 An exemplary method for dynamically visualizing the delivery of a device within a blood vessel is illustrated, which may be performed in Figure 1 The dynamic visualization may be used within a system of. The dynamic visualization may be, for example, a correlation or co-registration of a low-dose XA with one or more high-dose XAs. In some examples, the dynamic visualization may include a correlation of a live low-dose XA (e.g., a fluoroscopic image) with a high-dose XA (e.g., a cine angiography image). The high-dose image and the low-dose image may be correlated based on movement features detected in or derivable from the high-dose image and the low-dose image.

[0111] Dynamic visualization can occur in real time while low dose XA is being captured. Dynamic visualization can allow a physician to track the location of a device delivered in a blood vessel in real time on one or both of a low dose XA and a high dose XA. For example, dynamic visualization can allow a physician to track the location of a device relative to a vascular tree by providing an indication of the location of the device on a high dose XA (e.g., a vascular map). In some examples, dynamic visualization can allow a treatment zone identified on a high dose XA to be superimposed on a live low dose XA so that a physician can track the location of the device relative to the treatment zone.

[0112] In order to dynamically visualize the delivery of the device, a first extraluminal image 202 and a second extraluminal image 204 may be obtained. The first extraluminal image 202 may be a high-dose XA, and the second extraluminal image 204 may be a low-dose XA. According to some examples, a third extraluminal image may be obtained. The third extraluminal image may include one or more XAs obtained before the injection of the contrast agent. The XA obtained before the injection of the contrast agent may be a pre-contrast image. In some examples, the third extraluminal image may be generated or captured when the first extraluminal image 202 or the second first extraluminal image 204 is generated. For example, a sequence of XA images may be captured before the injection of the contrast agent. After the injection of the contrast agent, the sequence of XAs may continue to be captured. The XA captured before the injection of the contrast agent may be a pre-contrast XA, and the XA captured after the injection of the contrast agent may be a post-contrast XA. The post-contrast XA may include a low-dose XA and / or a high-dose XA.

[0113] The first extraluminal image 202 (eg, high-dose XA) can be used to generate a vessel map. The pre-contrast XA can be used to identify working vessels. The second extraluminal image 204 (eg, low-dose XA) can be used to detect and track the device.

[0114] The first extraluminal image 202 and the second extraluminal image 204 can be correlated based on movement features. The movement features can include, for example, a guide catheter tip, a distal endpoint of a working vessel, an i-th sampling point of the working vessel, an optical flow at an i-th sampling point of the working vessel, a translation of the working vessel, an optical flow at the guide catheter tip, an optical flow at a distal endpoint of the working vessel, a length of a cable tip, a cable tip at a regional location, an optical flow at a cable tip at a regional location, a proximal endpoint of the cable tip, a distal endpoint of the cable tip, an optical flow at a proximal endpoint of the cable tip, an optical flow at a distal endpoint of the cable tip, or the like. According to some examples, some movement features can be detected in one or both of the first extraluminal image 202 and the second extraluminal image 204. For example, the guide catheter tip, the distal endpoint of the working vessel, the i-th sampling point of the working vessel, an optical flow at an i-th sampling point of the working vessel, a translation of the working vessel, an optical flow at the guide catheter tip, and an optical flow at a distal endpoint of the working vessel can be detected in the first extraluminal image 202. In some examples, the guide catheter tip, the distal endpoint of the working vessel, the optical flow at the guide catheter tip, the optical flow at the distal endpoint of the working vessel, the length of the cable tip, the cable tip at the regional position, the optical flow at the cable tip in the regional position, the proximal endpoint of the cable tip, the distal endpoint of the cable tip, the optical flow at the proximal endpoint of the cable tip, and the optical flow at the distal endpoint of the cable tip can be detected in the second extraluminal image 204.

[0115] The motion features can be used to match the motion phase between the first extraluminal image 202 and the second extraluminal image 204. For example, the motion features can include global vessel motion, vessel region of interest motion, and vessel pixel motion. Global vessel motion can be a two-dimensional translation vector that represents a two-dimensional translation relative to the first frame at the t-th frame. The vessel region of interest motion can be represented by the cable tip region motion. The vessel pixel motion can generally correspond to "n" sampling points on the working vessel. Absolute spatial information (e.g., x-coordinates and y-coordinates) and relative spatial information (e.g., optical flow) can be determined for global vessel motion, vessel region of interest motion, and / or vessel pixel motion.

[0116] According to some examples, the motion features detected from the first extraluminal image 202 and the second extraluminal image 204 can be used to spatially and temporally correlate the motion phase of the first extraluminal image 202 and the motion phase of the second extraluminal image 204. The correlation of the first extraluminal image 202 and the second extraluminal image 204 can allow the dynamic visualization of the device within the blood vessel using a minimum amount of contrast agent. Typically, in order to visualize the position of the device within the blood vessel, a contrast agent must be injected to be able to identify the contour of the blood vessel. Therefore, in order to consistently view the device during the delivery of the device within the blood vessel, it may be necessary to continuously inject contrast agents. A large amount of contrast agents may cause negative side effects. Therefore, by correlating the vascular map generated based on the first extraluminal image 202 with a live low-dose XA image (e.g., the second extraluminal image 204), the amount of contrast agent required to track the position of the device during the delivery of the device can be minimized.

[0117] Working vessel identification

[0118] Figure 3A An exemplary flow chart for identifying a working vessel and aligning a guide catheter tip to display device markings is illustrated. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0119] To identify the working vessel, an extraluminal image frame 302 may be received. In some examples, the image frame 302 may be a pre-contrast XA. The pre-contrast image frame 302 may be captured prior to delivery of the device. For example, the image frame 302 may be captured after insertion of a guide wire and / or guide catheter but prior to delivery of the device.

[0120] AI may be used to segment guide catheters and / or wires within image frame 302. AI may include a deep learning model (e.g., VNet model 304). VNet model 304 may be trained to predict multiple categories (e.g., guide catheters, wires, device markers, and background). VNet model 304 may receive first image frame 302 as input and may output multiple segmentations.

[0121] The multi-annotated segmentation output by the VNet model can be converted into a multi-annotated mask 306. The multi-annotated mask 306 can illustrate the working vessel region as a shape prior for subsequent processes. Figure 4As shown, the pre-contrast frame 402 can be used to create a multi-annotated mask or a binary mask 404. The binary mask 404 created using the pre-contrast frame 402 can be used as a shape prior for a post-contrast frame. The post-contrast frame can be an extraluminal image frame captured after the contrast agent has been injected, such that the extraluminal image frame includes at least some contrast agent. According to some examples, after the contrast agent is injected and the extraluminal image of the vessel is captured, the contrast agent can highlight the working vessel as well as the side branches. In some examples, the contrast agent may make it more challenging to detect the wire in the extraluminal image. Therefore, using the binary mask 404 created from the pre-contrast frame (e.g., Figure 4 Frame 402 or Figure 3A The shape prior of the guide wire and / or guide catheter obtained in frame 302) can allow segmentation of the working vessel in the post-contrast extraluminal image.

[0122] like Figure 4 As shown, the pre-contrast XA frame 402 may include a guide wire 406 and / or a guide catheter. The pre-contrast XA frame 402 may be processed so that a binary mask 404 is generated. The binary mask 404 may be a binary mask of the guide wire 406 in the pre-contrast XA frame 402. According to some examples, the binary mask 404 may be used to identify the guide wire in the working vessel in the post-contrast frame 410. For example, the binary mask 404 of the guide wire 406 may be overlapped, superimposed, or combined with the post-contrast frame, as shown in frame 408. The shape prior of the guide wire 406 (also referred to as the working cable) may be used to segment the working vessel in the post-contrast image.

[0123] Return to reference Figure 3A , the shape prior 308 can be provided as an input to a deep learning model 310. The deep learning model 310 can be, for example, a MaskTrack+VNet model. The MaskTrack+Vnet model can be trained to identify working vessels. The MaskTrack+Vnet model 310 can be a spatiotemporal model. According to some examples, the MaskTrack+Vnet model 310 can be a spatiotemporal encoder-decoder segmentation network, such as Figure 5A As shown, the MaskTrack+Vnet model 310 can be trained to predict the working vessels.

[0124] The MaskTrack+Vnet model 310 can be trained using multiple annotations, such as Figure 5B504 as shown. Pre-contrast XA 502A may be annotated 504 as shown in frame 502B. Annotations 504 used to train MaskTrack+Vnet model 310 may be a set of lines that follow the trajectory of the guide catheter and guide wire in the extraluminal image before contrast injection. In some examples, annotations 504 may correspond to the working vessel trajectory. For extraluminal images captured after contrast injection, the path that follows the working vessel line may be annotated. For example, post-contrast XA 506A may be annotated 508 as shown in frame 506B. Annotations 508 may correspond to the working vessel.

[0125] When executing the MaskTrack+Vnet model 310, the input to the MaskTrack+Vnet model 310 can be an image frame and prediction information from one or more previous frames. For example, if the current frame is frame "t", the input to the MaskTrack+Vnet model 310 can be frame "t" and prediction information from previous frames "tn", "t-n+1"... "t-1" when "n" is greater than 1. According to some examples, the shape prior 308 of the previous frame can be an additional input to the MaskTrack+Vnet model 310. The shape prior propagation can correspond to the "MaskTrack" of the MaskTrack+Vnet model 310. The "VNet" of the MaskTrack+Vnet model 310 is a type of encoder-decoder segmentation network. However, other segmentation networks can also be used as part of the shape prior propagation workflow.

[0126] Using shape prior 308 can allow MaskTrack+Vnet model 310 to learn or predict the anatomical context of the working vessel with respect to the entire vascular tree. According to some examples, MaskTrack+Vnet model 310 (e.g., spatiotemporal segmentation model) may not be limited by previous frames. On the contrary, in some examples, shape prior 308 can be formulated as previous continuous frames based on the data or image to be processed. In addition to the first frame 302, MaskTrack+Vnet model 310 can also be executed using multiple subsequent frames 324 as input. Subsequent frames 324 can be frames captured after capturing the first frame 302. In some examples, subsequent frames 324 may include contrast agents. For example, subsequent frames 324 may be any combination of high-dose XA or low-dose XA, only low-dose XA, or only high-dose XA. Based on all frames (e.g., first frame 302 and subsequent frames 324), the output of MaskTrack+Vnet model 310 may be a vascular map 316 of the working vessel.

[0127] Fig. 22A method of determining device marker candidates from an extraluminal image and shape priors is illustrated. A device marker candidate can be a radiopaque marker captured in an extraluminal image. A device marker candidate can be associated with a marker of an intravascular device or an anatomical structure of a subject S. A marker candidate can later be confirmed as being associated with an intravascular device if certain criteria are met (e.g., following a path along a working vessel, moving at a speed consistent with the retraction of an intravascular device, being within range of an estimated position of an intravascular device, etc.). A device marker candidate can be confirmed as a device marker by a user or by a trained neural network.

[0128] The device marker candidate points 2241, 2242, and 2243 shown in frame 2203 can be extrapolated from the low-dose XA image 2201. A portion of the working vessel in the extraluminal image (e.g., the pre-contrast high-dose XA frame 2202) can be used to guide the extrapolation of the device marker candidate points 2241, 2242, and 2243. In this case, the working vessel can be used as a shape prior. To determine the initial shape prior, a pre-trained neural network can be used. Various methods (e.g., rigid body registration and elastic registration) can be applied to the initial shape prior to change its shape and increase its diversity. To detect device marker candidates, another neural network can be used. In some examples, the neural network can be a Vnet convolutional neural network. In another example, similar to Figure 3A The MaskTrack+Vnet model 310 described in the present disclosure can be used to detect device marker candidates. According to various aspects of the present disclosure, a neural network can be trained based on annotated marker positions of intraluminal devices detected on extraluminal images. According to various aspects of the present disclosure, the extraluminal images can be high-dose XA and low-dose XA. High-dose XA and low-dose XA can be used for training data in a transfer learning strategy. For the transfer learning strategy, high-dose XA and low-dose XA can be separated.

[0129] Fig.23 Two examples of cleaning shape priors are illustrated. In the first set of frames, frames 2301A and 2301B involve direct annotation of a working vessel 2304 from a binary mask frame 2301A. In the second set of frames, frames 2302A and 2302B involve inference of a second working vessel 2314 using a pre-trained SVS-Net model. The illustrated masks can be derived from an extraluminal image using a pre-trained model as described herein.

[0130] The shape priors used by the MaskTrack+Vnet model may require additional annotation. According to some examples, the shape priors for high-dose extraluminal images may be obtained by directly annotating the devices from the same images. In some examples, the shape priors for low-dose extraluminal images may be obtained by directly annotating the intraluminal devices in pre-contrast, high-dose XA during the same PCI procedure in which the low-dose XA was obtained. In other examples where pre-contrast XA is not present, the shape priors may be directly annotated on the working vessel in post-contrast, high-dose XA. In some examples, the annotation of the working vessel in post-contrast XA may involve another pre-trained neural network (e.g., the SVS-Net neural network shown in mask frame 2302A). Fig.23 In the example of FIG. 2 , the vessel tree mask frame 2302A is extrapolated from the post-contrast high-dose XA. In some examples, the post-contrast high-dose XA or its vessel tree mask 2302A can reference the low-dose XA containing device markers to refine the mask for the working vessel, which will be used as the working vessel in the shape prior.

[0131] The trained model can be applied to low-dose XA from the treatment phase of PCI to detect candidate markers. In some examples, the probability map output from the neural network model can be thresholded, and candidate marker locations can be extracted by finding the center point of each connected component in the thresholded image.

[0132] Fig.24 Annotations of shape priors for low-dose XA training data acquired from PCI under different conditions are illustrated. Frames 2401A, 2401B, and 2401C illustrate examples with pre-contrast high-dose XA. Frames 2402A, 2402B, and 2402C illustrate examples with post-contrast high-dose XA only. Frames 2401A and 2402A depict low-dose XA images. Frames 2401B and 2402B depict high-dose XA images. Frames 2401C and 2402C depict annotated working vessels 2404 and 2414, respectively, depicted on high-dose frames. Annotated working vessels 2404 and 2414 will be used as shape priors.

[0133] Referring back to FIG. 3 , the guide catheter and cable tip may be segmented out of subsequent frames 324. According to some examples, the guide catheter and cable tip may be segmented out of subsequent frames using the MaskTrack+Vnet model 310. In a common percutaneous coronary intervention (PCI) workflow, subsequent image frames 324 are not recorded. The number of recorded subsequent frames 324 (e.g., low-dose XAs) is typically less than the number of high-dose XAs captured during the diagnostic phase. In order to address the problem that the number of captured low-dose XAs is typically reduced, the system may enhance the high-dose XAs to generate enhanced low-dose XAs 322, so that the enhanced low-dose XAs 322 may be used as input to the training process for the MaskTrack+Vnet model 310 in the example of guide catheter and cable tip segmentation.

[0134] According to some examples, at least two factors may be applied to contrast and signal-to-noise ratio ("SNR") to convert a high-dose XA into an enhanced low-dose XA 322, which may be used to train a MaskTrack+Vnet model 310 or similar model to segment guide catheters and cable tips. High-dose XA captured prior to delivery of a device or during a PCI workflow may be processed to extract a neighborhood of a target object (e.g., a guide catheter, a cable tip, or the like). In some examples, the high-dose XA may be processed using dilation to extract a neighborhood of the target object. An intensity contrast of the target object in the high-dose XA may be determined. In some examples, a signal-to-noise ratio ("SNR") may be determined based on the high-dose XA and a low-dose subsequent frame 324. According to some examples, an enhanced low-dose XA 322 may be generated using a generative adversarial network ("GAN") type model or other generative type model.

[0135] The MaskTrack+Vnet model 310 may be trained using the enhanced low-dose XA 322 and the captured low-dose XA (e.g., subsequent frame 324) as input. The MaskTrack+Vnet model 310 may be trained to automatically detect guide catheters and cable tips. The MaskTrack+Vnet model 310 may be trained to predict the segmentation of guide catheters and cable tips. For example, when executed, the MaskTrack+Vnet model 310 may receive the subsequent frame 324 and provide a frame 312 as an output, wherein the frame 312 may be a segmented guide catheter and / or cable tip.

[0136] In some examples, frame 312 may be a skeleton frame. Skeleton frame 312 may be analyzed 326 for reference point propagation. Figure 3BAs shown, in addition to the working vessel 350, the skeleton frame 312A may also include one or more side branches 352. According to some examples, as shown in frame 312B, in order to keep the shape prior substantially similar to the working vessel 350, a skeleton analysis 326 may be performed. The skeleton analysis 326 may include, for example, a skeleton path search, side path removal, etc. In some examples, the output from MaskTrack+Vnet 310 is a working vessel map obtained after contrast agent is injected into the vessel. The guide catheter tip may not be visible due to occlusion caused by contrast X-ray attenuation. The shape prior from the frame 306 showing multiple annotations can be aligned with the skeleton analysis 326 results using a geometric transformation. After alignment, as shown in FIG. Figure 3B As illustrated in frame 312C in FIG. 306 , the guide catheter tip in frame 306 (indicated by circle 354 ) may be projected to the closest neighboring skeleton point as the propagated guide catheter tip in XA injected with contrast agent (indicated by triangle 356 ).

[0137] According to some examples, frame 324 may be used to detect and track 318 the location of a marker on a device delivered within a blood vessel. An AI model, a convolutional neural network, or the like may be used to detect and track 318 the location of a marker on the device. Details on how to detect and track the marker will be referred to in Fig.12 and Fig.13 Discuss in more detail.

[0138] For high-dose XA, MaskTrack+Vnet 310 can be used to obtain a working vessel map 316 and guide catheter tip position, such as Figure 3C For example, the guide catheter tip position 366 can be automatically identified by MaskTrack+Vnet 310 along the curve passing through the main vessel 364. Using a similar method, for live low-dose XA, the guide catheter and its tip position 366' can be identified, as shown in FIG. Figure 3C 362 in the image. The device marker 368 in the live XA (e.g., frame 362) can be detected and tracked. In some examples, the imaging pose between the vessel map and the live low-dose XA may be slightly different. In such an example, the image background between them may not be aligned, as illustrated in frame 370. To address this issue, the guide catheter tips 366, 366' in both the vessel map 360 and the live XA 362 can be aligned by translating one image frame to another image frame so that the image background can be correctly blended together, as illustrated in frame 372. The marker 368 can then be displayed on the working vessel using the nearest neighbor point or other geometric projection for the marker display 320, as illustrated in frame 374.

[0139] In some examples, a region of interest can be determined using device marker candidates detected along the working vessel. A candidate region of interest (cROI) can represent a box containing the center points of one or more device marker candidates detected by the neural network model. According to some examples, for each device marker candidate detected within a frame of low-dose XA, a cROI box can be determined, whose center point is set to the center point of the detected device marker candidate. In order to reduce the number of cROI boxes without losing track of the device markers, various methods (e.g., a non-maximum suppression step (NMS)) can be applied. In another example, the cROI box can be determined from the working vessel. In some examples, the working vessel can be determined in a pre-contrast, high-dose XA and overlaid on a live feed of a low-dose XA.

[0140] Fig.25 A method of determining a region of interest based on detected device marker candidates is illustrated. The method may include matching the detected device marker candidate with the closest labeled cROI along the working vessel. This method of pairing the detected device marker candidate with the cROI serves as a confirmation step to ensure that the tracked device marker falls along the working vessel. Fig.25 In the example, the working vessel 2504 defined by the shape prior can be represented by a plurality of spatially continuous boxes (including boxes 2521, 2522, 2523, and 2524). These boxes can be the same size. In some examples, the boxes can be of different sizes to fit along the length of the working vessel 2504. In some examples, the size and number of boxes can be input by a user or automatically selected by an AI algorithm. The boxes can be associated with device marker candidates (e.g., device marker candidates 2541, 2542, and 2543) detected along the path 2531 of the intravascular device. The device marker candidate can be associated with the closest one of the number of continuous boxes. For example, device marker candidates 2541, 2542, 2543 can be associated with boxes 2522, 2523, and 2524, respectively. In some examples, boxes, such as box 2521, can be retained when the device continues to move along the path 2531 over time during the treatment phase. At different times during the treatment phase, the device marker can fall into different areas of interest. This method can keep a constant number of regions of interest and update them based on each frame of the low-dose XA image, as described below. Fig.26 Alternatively, in some examples, a box in which no device tag candidate is detected may not qualify as a region of interest, for example, box 2521.

[0141] The device marker candidate points can be the center of their associated cROI. The initial working vessel trace points are determined from the working vessel detection 2504 of the high-dose XA image. Given the working vessel detection 2504 from the high-dose XA image used as a shape prior, the trace points can be sorted along their path using a breadth-first search (BFS) algorithm. The cROI associated with each detected marker with the smallest overlap and closest to it can be selected along the working vessel tracking in the new frame of the low-dose XA image. In some examples, the device marker candidate may not be aligned with the center of the closest box. In this case, the AI ​​model can estimate the center point closest to the device marker candidate along the path of the vessel as the center of the box. For example, the device marker candidate 2542 is closest to the box 2523. Because the device marker candidate does not fall at the center of the box 2523, the model can create a center point 2544 closest to the device marker candidate 2542. Fig.26 The Kalman filter in solves the problem that the device marker candidate is not aligned with the center of the closest box. At the end of the algorithm in box 2612, the center of the region of interest is located near the device marker candidate. In some examples, the cROI containing the device marker detected from the neural network model can be determined as the region of interest. In another example, the cROI box containing the centroid within a predetermined distance from the device marker detected from the neural network model can be determined as the region of interest. As long as the device marker is within a given distance to the centroid of the box, the device marker can fall outside the box.

[0142] In another example, if a cROI or box passes a persistence criterion, it can be determined to be a region of interest. In some examples, the persistence criterion can be to check whether there are more than 5 low-dose XA frames with the region of interest in the past and present 9 consecutive frames. In some examples, if the cROI is located at the distal end of the guide catheter tip, the cROI can be determined to be a region of interest. In the case of contrast injection during the treatment stage, the distribution of pixel values ​​within a given region of interest can be compared with the distribution of pixel values ​​from the same region of interest in past frames. In some examples, frames with contrast injection can be detected across the low-dose XA series.

[0143] A major limitation of existing approaches is the need for device-specific templates to automate the detection and tracking of device markers. For advanced treatment techniques that require the delivery and coordination of multiple devices (e.g., kissing balloons), each device may need to be registered separately before use, complicating the overall workflow of the procedure. This fully automated approach improves the real-time visualization of device markers in low-dose XA by detecting regions of interest containing device marker candidates within each frame of the live feed of low-dose XA. The approach is fast and highly adaptable to changes in the treatment workflow, thereby expanding the usability of dynamic visualization of device markers to help physicians navigate complex clinical scenarios. The approach can also be extended to other applications that require device marker visualization (e.g., electrophysiology (EP), pulmonary and peripheral angiography applications, or other imaging modalities involving image guidance (e.g., MRI or IVUS)).

[0144] Fig.26 A flow chart of an exemplary workflow for tracking regions of interest in two subsequent frames of low-dose XA is illustrated. According to some examples, each region of interest may be represented by Fig.25 The corresponding box center point depicted in . The box may have a fixed width parameter and a fixed height parameter. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0145] In box 2600, the model can estimate the box center points of the candidate region of interest based on the extraluminal image. In some examples, a Kalman filter can be used to estimate the box center points. According to some examples, in order to reduce false positives, after the prediction step of each Kalman filter iteration, the predicted marker position can be compared with the detected marker position 2601. In the prediction step, according to some examples, the loss function between the pixel values ​​in each box of two subsequent frames from the low-dose XA image and the pixel values ​​in the adjacent boxes of the same size can be calculated. In some examples, the loss function can be a 2-D cross-correlation. In another example, the loss can be some other similarity or dissimilarity criteria for image registration. The displacement between the two boxes with the minimum loss function (dissimilarity criteria, or the negative value of similarity criteria) can be used to predict the movement of the box. According to another example, the displacement can be calculated using optical flow to predict the movement of the box. In some examples, the Lucas-Kanade (LK) method can be used to calculate the optical flow.

[0146] In block 2602, the model may determine pairwise distances between predicted box center points and detected device marker center points in the current frame. The model detects device marker candidate locations on the extraluminal image.

[0147] In block 2604, the model can identify each newly detected device marker center point and pair it to its closest box. In all pairs, the corresponding distances can then be compared to the width and height of the box to determine how close the device marker is relative to the closest box that has been paired.

[0148] In block 2606, the model can filter out pairs where the device marker is outside, or where the distance to its closest box is greater than a predetermined threshold, or where the device marker is inside the guide catheter. In this step, the model reduces the number of false positives detected by the model by removing any markers that are far from the box. False positives may be the result of foreign matter or the anatomical structure of the subject S.

[0149] In block 2608, it is determined whether pairs with markers that are close to the box need to update or adjust the box center point. In some examples, the detected marker may not be the natural center point of the closest box. In some examples, multiple markers are detected within the box, of which only one marker may be the natural center point of the closest box. This is allowed because the method can simply average all displacements into a constant displacement (see Fig.26 2610) to translate all trace points by the same amount in some cases. Thus, a one-to-one correspondence is not required. If no pairs exist after filtering, or no pairs need to be re-centered, the model can move to block 2614. If the pairs need to be adjusted to update the box center point, the model can proceed to block 2610.

[0150] In block 2614, there are no remaining pairs that need adjustment, so no position updates are made to the boxes in the current frame. In block 2610, if adjustments to the box center points are required, a Kalman filter update step may be performed. In this update step, the position of each detected marker may be used to update the Kalman prediction 2600 of its closest box center position, and the displacement between the closest box center points of each marker before and after the Kalman update may be averaged across all remaining pairs.

[0151] In block 2612, the model may update the box center point based on the determined average displacement.

[0152] 3 , when the delivery of the device in the vessel is dynamically visualized 314, a working vessel 316 may be provided for output as a vessel map. Once aligned 320, detected and tracked markers 318 may be provided for output. For example, markers may be provided for output on a vessel map, thereby providing an indication of the location of the device in real time without having to inject additional contrast agent. According to some examples, a treatment zone initially identified on a vessel map may be provided for display on a live low-dose XA image, so that the location of the device relative to the treatment zone may be seen on the low-dose XA. In some examples, markers may be provided for output on a vessel map, thereby providing an indication of the real-time location of the device relative to the treatment zone.

[0153] refer to Fig.15 , one or more markers along the working vessel can be identified. For example, the working vessel 1504 can be identified in frame 1502A. Two markers can be identified, marker 1 and marker 2. Markers 1, 2 can be automatically identified and / or identified based on the received input. For example, the system can receive input corresponding to the selection of the marker via a computing device, a touch screen, etc. Markers 1, 2 can correspond to a working area or a device landing area. The landing area can correspond to a coordinate relative to the catheter tip "C". In some examples, the distance between markers 1, 2 and the catheter tip "C" can be determined in each high-dose XA. For example, the distance between the catheter tip C and marker 1 can be "r pixels", and the distance between the catheter tip C and marker 2 can be "q pixels". Pixels along the working vessel trajectory within a distance range (e.g., the distance between the markers) can correspond to the determined landing area. As Fig.15 As shown, pixels in the landing zone can be determined as "q pixels" minus "r pixels". In some examples, the landing zone can be referred to as a treatment zone. According to some examples, the blood vessel may not be visible in the low dose XA. In such an example, when the guide catheter tip position "C" is available, combining the above pixel distance information can provide a treatment zone on the output. Providing a treatment zone on the output (e.g., on a display) can assist the user in the delivery of the device.

[0154] Multi-scale moving features

[0155] The XA taken during PCI usually presents a mixed movement of the patient's heartbeat and breathing. This movement makes it challenging to correlate the live XA image with the previously captured XA image. Further, there are usually inherent projection characteristics, that is, the blood vessels shown in high-dose XA and low-dose XA are two-dimensional projection geometries, rather than three-dimensional geometries. Therefore, accurately identifying the correspondence of the physical pixels of high-dose XA and low-dose XA may be challenging and impractical. However, different pixels can present different movement patterns, and the movement of different pixels can show similar patterns. Therefore, multi-scale movement features can be used to correlate high-dose XA and low-dose XA. By using multi-scale movement features, the process of correlating high-dose XA and low-dose XA can be completed without using external sensors (e.g., EKF), timestamps, or time and computationally intensive methods.

[0156] Figure 6 An exemplary method of correlating a live XA image (e.g., a low-dose XA or a fluoroscopic image) with a previously captured XA (e.g., a high-dose XA) based on a motion feature is illustrated. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0157] In order to correlate the motion phases of the high-dose XA and the low-dose XA, image features corresponding to the mixed motion may be identified. A multi-scale motion feature library 602 may be generated. The multi-scale motion feature library may provide various aspects of motion information and may be used in a subsequent phase matching process. The multi-scale motion features may include global vessel motion, vessel region of interest motion, vessel pixel motion, or the like.

[0158] 7 illustrates global vessel movement 702, vessel region of interest movement 702, and vessel pixel movement 706. Vessel level movement 702 may correspond to a two-dimensional translation vector. The two-dimensional translation vector may be represented as: Tx(t, 1) and Ty(t, 1). The two-dimensional translation vector may be a two-dimensional translation at the t-th frame relative to the first frame.

[0159] The vessel region of interest movement 704 may correspond to the cable tip level movement. The cable tip level movement may include, for example, absolute spatial information and relative spatial information. The absolute spatial information may correspond to coordinates (e.g., x-coordinates and y-coordinates). The relative spatial information may correspond to the optical flow between two adjacent frames.

[0160] The vessel pixel shift 706 may correspond to an "n" number of sampling points along the working vessel. According to some examples, the sampling points may be located between the tip of the guide catheter to the distal end of the vessel. The sampling points may be identified or detected using the VNet model 304 and / or the MaskTrack+VNet model 310. According to some examples, absolute spatial information and relative spatial information may be determined for the vessel level pixel shift.

[0161] Return to reference Figure 6 , the segmented guide catheter and cable tips obtained from low-dose XA during PCI can be used to derive multi-scale motion features. For example, the segmented cable tip can be used to annotate the cable tip level motion feature 704. In some examples, the working vessel can be sampled between the cable tip and the guide catheter tip to detect pixel level features 704. For example, for each pixel, the motion feature type can be derived in the form of a period. High-dose XA or low-dose XA can be used to derive or determine the pixel level 704 motion feature. The type of detected motion feature can be based on whether the feature is detected from high-dose XA or low-dose XA.

[0162] The movement features may include, for example, a guide catheter tip, a distal endpoint of a working vessel, an i-th sampling point of the working vessel, an optical flow at the i-th sampling point of the working vessel, a translation of the working vessel, an optical flow at the guide catheter tip, an optical flow at the distal endpoint of the working vessel, a length of the cable tip, a cable tip at a regional location, an optical flow at the cable tip in a regional location, a proximal endpoint of the cable tip, a distal endpoint of the cable tip, an optical flow at the proximal endpoint of the cable tip, an optical flow at the distal endpoint of the cable tip, or the like. According to some examples, some movement features can be detected in one or both of the high-dose XA and the low-dose XA. For example, the guide catheter tip, the distal endpoint of the working vessel, the i-th sampling point of the working vessel, an optical flow at the i-th sampling point of the working vessel, a translation of the working vessel, an optical flow at the guide catheter tip, and an optical flow at the distal endpoint of the working vessel can be detected in the high-dose XA. In some examples, the guide catheter tip, the distal endpoint of the working vessel, the optical flow at the guide catheter tip, the optical flow at the distal endpoint of the working vessel, the length of the cable tip, the cable tip at the regional location, the optical flow at the cable tip in the regional location, the proximal endpoint of the cable tip, the distal endpoint of the cable tip, the optical flow at the proximal endpoint of the cable tip, and the optical flow at the distal endpoint of the cable tip can be detected in low-dose XA.

[0163] When plotted, moving features can form periodic patterns. For example, Figure 7BAs shown, graph 700A is a representation of the x-translation of global vessel movement, and graph 700B is a representation of the y-translation of the vessel pixel movement of the sampling point along the working vessel. The horizontal axis of each graph corresponds to the frame number, and the vertical axis corresponds to the feature value. As shown, the movement features in graphs 700A, 700B have a substantially periodic pattern.

[0164] like Fig.11 As shown, the heartbeat cycle of the patient can be determined based on the mobile feature. The auto-correlation function ("ACF") can be applied to the multi-scale mobile feature 1102. For example, the mobile feature 1102 can be the original mobile feature converted to the ACF. In some examples, the mobile feature of the auto-correlation can be plotted to identify the peaks and troughs. For example, the signal associated with the mobile feature can be processed (e.g., by finding the derivative of the signal). According to some examples, the first-order derivative or the second-order derivative of the signal can be used to identify the peaks and troughs of the mobile feature. The distance between adjacent peaks can be determined. In some examples, the distance between adjacent troughs can be determined. The distance between the peaks and troughs can be voted 1106 in the binned space. According to some examples, the mobile feature may present some noise on the time axis. For example, the peak #1 in feature #1 may be located at the 10th frame, and the peak #1 in feature #2 may be located at the 11th frame. However, these features may point to being related and / or associated with the same heartbeat moment. As a result, the time space can be binned into a coarser resolution. For example, the 10th frame and the 11th frame can be combined into one bin, and two votes can be put into this bin. Thus, the votes for the heartbeat time can be concentrated and enhanced.

[0165] After voting 1106, NMS 1108 may be applied. NMS 1108 may refine the heartbeat period in both the peak domain and the trough domain. In some examples, the voting result may still present multiple peaks near the heartbeat time, depending on the merge size. NMS may be performed to suppress noisy votes and retain the maximum votes in the local time neighborhood. As a result, the heartbeat time and period may be extracted more reliably.

[0166] A filter 1110 (eg, a median filter or other statistical filter) may be used to identify the cardiac cycle. The cardiac cycle may be used when correlating the shift phases of the low-dose XA and the high-dose XA.

[0167] Spatiotemporal phase matching

[0168] review Figure 6, the live multi-scale motion features can be used to correlate the motion phases of high-dose XA and low-dose XA in time. The motion feature library 602, the live multi-scale motion features 606, and the frame rate alignment 608 can be used to correlate 612 high-dose XA and low-dose XA in time. Correlating high-dose XA and low-dose XA in time can include, for example, identifying a corresponding high-dose XA for each low-dose XA. The corresponding high-dose XA can have a substantially similar temporal consistency to the low-dose XA. According to some examples, the high-dose XA can be generated or captured before the delivery of the device. The temporal consistency can be based on, for example, when the low-dose XA and the high-dose XA are captured during the cardiac cycle. According to some examples, the temporal consistency in the low-dose XA and the high-dose XA can be determined by comparing the cardiac motion in the corresponding images. Over time, the cardiac motion can repeat itself, so that the cardiac motion observed in the low-dose XA sequence can look similar to the cardiac motion observed in the high-dose XA sequence. The similarity between the cardiac motion in the low-dose XA sequence and the high-dose XA sequence can be determined or quantified using a representative moving feature. For example, the temporal evolution of the representative moving feature can be correlated and used to determine the cardiac motion. For example, in some examples, the cardiac motion in the low-dose XA and high-dose XA can be determined based on the position of the tip of the wire or the position of the tip of the guide catheter. In another example, the cardiac motion can be determined based on the displacement of the tip of the wire or the tip of the guide catheter over a sampling time interval of the angiographic imaging system, or by any other similar physical mechanism, or a combination thereof. In some examples, if the frame rates during the acquisition of the high-dose XA and low-dose XA are different, the moving feature can be used to align the frame rates 608.

[0169] Figure 8 An exemplary method for temporally correlating the shift phases of high-dose XA and low-dose XA is illustrated. To temporally correlate the high-dose XA and low-dose XA, an initial search is performed. In some examples, after performing the initial search 802, an online update 812 is performed. According to some examples, the initial search 802 can identify results for correlation for subsequent online updates 812.

[0170] Fig. 9 An exemplary method for performing an initial search is illustrated. An initial search may be performed to determine an initial time shift. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0171] In block 902, extraluminal images (eg, low-dose XA and high-dose XA) are obtained. Motion features may be derived from the high-dose XA and the low-dose XA, respectively.

[0172] In block 904, a motion feature may be selected from the high dose XA and the low dose XA. According to some examples, a plurality of motion features may be selected from both the high dose XA and the low dose XA. In some examples, the motion features may be resampled at a common frame rate. In some examples, the frame rate may be used to estimate a time interval between incoming low dose XAs during a subsequent online update 812.

[0173] A correlation coefficient may be determined for the mobile feature in the high-dose XA and the low-dose XA. The correlation coefficient may be determined using a normalized cross correlation ("NCC"). For example, the NCC may be determined using the following equation:

[0174]

[0175] In the above equation, X and Y are the high-dose XA motion characteristics and the low-dose XA motion characteristics, X k and Y k is the kth sample of these moving features, and is the average value of these moving features in their corresponding time windows.

[0176] For a given pair of moving features, samples of low-dose XA features that fall into a fixed time window can be identified. The window can be compared with all time windows of the same length from the corresponding high-dose XA feature. Between each time window from the high-dose XA feature and the time window from the low-dose XA feature, a time shift can be identified. In some examples, the time shift can be identified as the order of each time window of the high-dose XA feature relative to the full time window of the same high-dose XA feature. In some examples, the order of the time window of the high-dose XA feature can be determined by the order of its last feature sample relative to its full time window. The comparison can be performed by the correlation coefficient in the NCC equation. The process can be performed for one or more combinations of moving features from high-dose XA and low-dose XA to determine a pair of high-dose XA features and low-dose XA features with the maximum correlation coefficient. According to some examples, the process can be performed for one, some, most or all combinations of moving features from high-dose XA and low-dose XA to determine a pair of high-dose XA features and low-dose XA features with the maximum correlation coefficient. In some examples, the optimal time shift may be determined as the time shift between the time window of the high-dose XA signature and the time window of the low-dose XA signature having the largest correlation coefficient.

[0177] The NCC may be used to identify the mobile features in the low-dose XA and high-dose XA pairs that have the greatest cross-correlation. For example, in block 906, the correlation coefficient may be compared to a threshold. If the correlation coefficient is below the threshold, the initial search may be indicated as a failure 918. If the correlation coefficient is above the threshold, the initial search may continue. According to some examples, the threshold may be between about 0.6-0.8.

[0178] In block 908, the motion features used to determine the correlation coefficient may be compared. If the motion features in the low dose XA and the high dose XA are the same type of motion features, an initial search may be completed and an initial time shift may be determined 916. If the motion features in the high dose XA and the low dose XA used to determine the correlation coefficient are different types of motion features, the initial search may proceed to block 910.

[0179] In box 910, a second moving feature in the high-dose XA moving feature can be selected. The second moving feature in the selected high-dose XA moving feature can be of the same type as the moving feature of the selected low-dose XA. The first feature and the second feature of the selected high-dose XA can be used to determine the phase shift offset caused by the mismatch in the moving feature type. Compared with the high-dose XA, the low-dose XA generally includes less information. To take this into account, the initial search 802 expands the feature pairs that can be used so that the success rate of the initial phase correlation increases. In some examples, the estimated time shift may include additional drifts in examples where the high-dose XA features and the low-dose XA features are different. The additional drift may bias the results of the mobile phase correlation. To take this bias into account, a second NCC can be used to estimate the potential drift between the selected high-dose XA features, and then offset according to all time shifts estimated in the initial search 802 and the online update 812.

[0180] In block 912, a cross-correlation coefficient or correlation coefficient of the selected high-dose XA motion features may be determined. A second NCC may be used to determine the cross-correlation. In some examples, the second NCC may be the same as the NCC equation provided above, or may be a different NCC. In addition to the NCC, other metrics (e.g., various distance-based metrics) may also be used to correlate the high-dose XA and the low-dose XA in time, or to estimate phase shift offsets between different types of motion features as in blocks 904 and 912.

[0181] In block 914, a correlation coefficient determined using a second NCC (e.g., a second correlation coefficient) may be compared to a threshold. If the second correlation coefficient is below the threshold, the initial search may be indicated as a failure 918. In such an example, a second search including a subset of features from block 904 may be performed. In some examples, the second search may use only high-dose XA features and low-dose XA features of the same type, thereby skipping blocks 910-914. If the second correlation coefficient meets or exceeds the threshold, an initial time shift 916 may be determined. At each of blocks 904 and 912, a time shift that maximizes the corresponding NCC may be obtained. The initial time shift may then be determined as the difference between the two time shifts (e.g., the time shift at block 904 and the time shift at block 912). According to some examples, the time shift determined according to the second NCC may be used as an offset and subtracted from the time shift determined according to the first NCC. In some examples, the initial time shift accepted in block 908 may be a case where the time shift from block 912 is 0. Figure 8 The online update 812 shown in can iteratively filter the initial time shift.

[0182] According to some examples, the high-dose XA shift feature may have a limited sample size. A cardiac cycle may be determined for initial search, and the high-dose XA within the cycle may be used and / or included as a plot for dynamic visualization. The start time of the cardiac cycle may be determined as the initial time shift. In some examples, the length of the cycle may be determined as a heartbeat cycle, such as with respect to Fig.11 described.

[0183] Fig.10 An exemplary method for performing an online update is illustrated. An online update may be performed to determine a time shift. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0184] In block 1002, an initial time shift is received. The initial time shift may be denoted as "n-1". The online update may be performed iteratively. For example, the online update may take the time shift obtained at time n-1 and estimate the time shift at time n. "n" may represent the nth low-dose XA motion signature sample during PCI. In some examples, the initial time shift may correspond to n=1.

[0185] According to some examples, the maximum NCC between the low-dose XA feature and the high-dose XA feature may decrease over time, which may result in errors in matching the low-dose XA to the high-dose XA. In some examples, the ongoing process of low-dose XA acquisition may result in the number of high-dose XAs available for correlation being exhausted if the previous high-dose XAs are not used again. To account for this, a single cardiac cycle of high-dose XA frames may be used when correlating high-dose XAs and low-dose XAs in time. Then, low-dose XAs from the same phase of different cardiac cycles may be correlated with the same high-dose XAs taken in a single cardiac cycle.

[0186] In block 1004, it may be determined whether the new low dose XA is outside the mapping period. For example, for each new low dose XA sampled during PCI, if the time shift (e.g., "n-1") is greater than the initial time shift by more than one heartbeat period, the low dose XA may be considered to be outside the mapping period. If the new low dose XA is outside the mapping period, the online update may proceed to block 1006. If the new low dose XA is not outside the mapping period, the initial time shift may be input to block 1008 for filter prediction.

[0187] In block 1006, the time shift may be reset by offsetting the heartbeat period from time shift "n-1". In some examples, other criteria may be used to reset the time shift. This time shift may then be input to block 1008 for filter prediction.

[0188] In block 1008, a filter may be used to predict a new time shift. In some examples, the filter may be a linear Kalman filter. A new time shift may be proposed to chronologically correlate a new low-dose XA frame with the next available high-dose XA frame. The new low-dose XA frame may be, for example, the most recently acquired low-dose XA frame.

[0189] In block 1010, the sample window may be updated. In some examples, the sample window may be updated by removing the earliest low-dose XA signature sample from the last sample window and adding the most recent low-dose XA signature sample to the last sample window. The optimal time shift may be the time shift that maximizes the NCC, which is detailed in 1014. The time shift correction is detailed in 1018.

[0190] In block 1014, the time shift is refined based on the updated sample window using cross-correlation.For example, the time shift may be refined by shifting the time window of the low-dose XA shift feature until a maximized NCC is identified.

[0191] The cross-correlation may generate a correlation coefficient. In block 1016, the correlation coefficient may be compared to a threshold. If the correlation coefficient is below the threshold, a time shift may be determined 1020 without updating the filter. If the correlation coefficient meets or exceeds the threshold, the filter may be updated 1018.

[0192] In block 1018, when the correlation coefficient meets or exceeds a threshold, the filter may be updated. The filter may be, for example, a linear Kalman filter. According to some examples, the following Kalman filter equation may be used to update the time shift: in is the predicted new time shift (block 1008), z n is the time shift optimized by the NCC (block 1014), and K n is the Kalman gain that balances the predicted value and the NCC measurement value. Filter update 1018 can be used to reduce the uncertainty of the prediction.

[0193] In block 1020, the determined time shift may be provided after the filter has been updated.

[0194] Back to reference Figure 6 , the live multi-scale motion features can be used to spatially correlate the motion phases of high-dose XA and low-dose XA. The motion feature library 602, the identified working vessels 614, the live multi-scale motion features 606, and the device segmentation 610 can be used to spatially correlate the high-dose XA and the low-dose XA. In addition to temporal correlation, as an alternative to or refinement of temporal correlation, spatial correlation can also be used. For example, the spatial fit between the spatial landmark of the low-dose XA and the working vessel of the high-dose XA can be evaluated and used to refine the temporal correlation of the high-dose XA and the low-dose XA.

[0195] AI-based methods can also be used to capture the intrinsic motion states of the heart vessels in high-dose XA and low-dose XA. In some examples, these intrinsic motion states can be captured directly from XA by extracting deep features using convolutional neural networks. In addition to multi-scale motion features, these deep features can also be used as a substitute or refinement of multi-scale motion features to correlate high-dose XA images with low-dose XA images using time-based, spatial-based indicators or a combination thereof.

[0196] According to some examples, for each low-dose XA, given the associated high-dose XA, a loss function may be determined. In some examples, the loss function may be a distance transform value. If the correlation between the high-dose XA and the low-dose XA is substantially accurate, the spatial loss may be a low value compared to the spatial loss value when the correlation between the high-dose XA and the low-dose XA is inaccurate. The low loss value may correspond to the correlation of the spatial landmark and the working vessel. If the spatial mismatch is significant (e.g., the loss value is high), the correlation between the high-dose XA and the low-dose XA may be determined to be inaccurate.

[0197] In some examples, when the correlation between the high-dose XA and the low-dose XA is inaccurate, the system can automatically provide corrections. For example, the system can provide outputs of correlation frames from the most recent, accurate correlation. In some examples, the system may not display the current frame due to inaccurate correlation. In another example, the system can automatically perform spatial registration to identify correlated phase frames.

[0198] According to some examples, one or more markers along the working vessel can be identified. The marker can be automatically identified and / or recognized based on the input received. For example, the system can receive input corresponding to the selection of the marker via a computing device, a touch screen, etc. The marker can correspond to a working area or a device landing area. The landing area can correspond to the coordinates relative to the catheter tip. In some examples, the distance between the marker and the catheter tip can be determined in each high-dose XA. Pixels along the working vessel trajectory within a distance range (e.g., the distance between the markers) can correspond to the determined landing area.

[0199] The spatial correlation and / or temporal correlation of high-dose XA and low-dose XA can be fused 618 to provide dynamic visualization 620 of the delivery of the device within the blood vessel. For example, the system can provide output spatially correlated and / or temporally correlated low-dose XA and high-dose XA. The output can include any of a blood vessel map, a live tracking of the position of the device, an indication of the treatment zone, an indication of the position of the device relative to the treatment zone, or the like. In some examples, the output can include multiple images, videos, video screenshots, or the like. For example, two or more images and / or videos can be output relative to each other. One image can be a blood vessel map, and another image can be a live image of the position of the tracking device. In some examples, the output can allow the doctor to use less contrast agent to dynamically visualize the delivery of the device within the blood vessel. Since the dynamic visualization is associated with high-dose XA and low-dose XA, the dynamic visualization can use less contrast agent than having to continuously inject contrast agent to enable the visualization of the blood vessel.

[0200] Marker detection and tracking

[0201] Device marker detection and tracking can allow for dynamic visualization of devices during the treatment phase of PCI. Devices can be represented by their unique radiopaque markers. Detection and tracking of markers can be performed by iteratively estimating the position and movement state of markers in low-dose XA. Iteratively estimating marker positions allows detection and tracking to be performed accurately and quickly. In addition, the process can quickly recover from errors.

[0202] According to some examples, the device can be a balloon device. However, the device can be any device, such as an imaging probe, a pressure wire, a stent delivery device, a vascular preparation device (e.g., an atherectomy device, or a lithotripsy device, or the like). Therefore, the example provided herein about a balloon device is only an example and is not intended to be limiting. The balloon device provided in this example can be replaced with any device having a radiopaque marker or a marker visible in XA.

[0203] Fig.12 is an exemplary method for detecting and tracking markers on a device. A low-dose XA 1202 may be captured. In some examples, a live low-dose XA 1012 may be captured (eg, during delivery of the device).

[0204] To predict the position of the marker in the corresponding low-dose XA, the movement of the marker in the most recent sampling period can be estimated using optical flow estimation 1208. Optical flow assumes that the local intensity of moving objects in the image remains approximately constant at least for a short duration. According to some examples, the optical flow equation can be: I( x ,t)≈I( x +δ x ,t+δt), where δ x is the local image area at position x The displacement from time t to t+δt at t is given by the above equation. The above equation can be applied to each pair of low-dose XA sampled at intervals of δt. By applying the optical flow equation to the position of each detected marker, δ x can be determined and used as a marker movement. According to some examples, δ x It can be determined from the optical flow equation using the iterative Lucas-Kanade (ILK) method. In some examples, δ x May contain a horizontal motion component u and a vertical motion component v. The motion estimate may be used to predict the position of the marker in the nearest low-dose XA. According to some examples, a filter (eg, a linear Kalman filter 1210) may be used to predict the position of the marker based on the motion estimate.

[0205] In order to update the position of the marker in the corresponding low-dose XA, a neural network (such as a V-Net convolutional neural network) can be used to detect marker candidates from the same low-dose XA. The V-Net marker detector 1204 can be trained using the annotated XA. The XA can be annotated to identify the marker position of the device within the XA. According to some examples, the V-Net marker detector 1204 can be trained based on the annotated marker position of the intraluminal imaging probe (e.g., OCT probe) in the XA. Additionally or alternatively, the marker detector 1204 can be trained based on the annotated marker position of another device (e.g., a balloon delivery device) in the XA. Training the marker detector 1204 based on the marker position of any type of device (e.g., OCT probe) can allow the marker detector 1204 to detect markers from different types of devices (e.g., balloon delivery devices). According to some examples, high-dose XAs and low-dose XAs in the training data can be separated to adopt a transfer learning strategy. The marker detector 1205 can be trained to predict a probability map measuring the membership of each pixel to a marker class. According to some examples, the V-Net model can predict the probability that each location on the low-dose XA image is from a marker. The "membership" can correspond to whether the location on the XA image is from a marker. The higher the probability value (in the range of 0-1), the more likely the location on the XA is from a marker.

[0206] According to some examples, when executing the marker detector 1204, the marker detector 1204 can receive as input the low dose XA 1202. The marker detector 1204 can output a probability map measuring the membership of each pixel to a marker class. According to some examples, the output of the marker detector 1204 can be processed to integrate model inference into the Kalman filter 1210.

[0207] The processing may include detecting center points 1206. The center points may represent detected markers. For example, the probability map output by the marker detector 1204 may be thresholded to segment candidate marker regions. The center point of each candidate region may be extracted. The extracted center point of each region may correspond to a detected marker.

[0208] Fig.13 An exemplary method for tag tracking using V-Net model inference is illustrated. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted. Fig.13 can be understood as a combination Fig.26 An alternative method to the method described. Specifically, Fig.13 A detection and tracking method is described which aims to obtain the accurate position of a device marker, and Fig.26A detection and tracking method aimed at increasing the local image contrast between device markers and the background is described.

[0209] According to some examples, to reduce false positives, after the prediction step of each Kalman filter iteration, the predicted marker positions 1302 can be compared to the detected marker positions 1304 determined by the marker detector 1204. A unique candidate marker position can be assigned for the comparison. According to some examples, to determine the assignment of the marker position, pairwise distances 1306 can be evaluated based on a set of predicted and detected marker positions 1302, 1304. In some examples, the pairwise distances can be Euclidean pairwise distances.

[0210] The predicted positions may then be sorted 1308 in ascending order based on their distance to the closest detected position. Filter updates (e.g., Kalman filter updates) may be iteratively performed 1310 in the sorted order. Filter updates are performed on the predicted positions using the closest detected positions. To avoid conflicts, already assigned detected marker positions and those in the surrounding area may not be considered for new assignments.

[0211] If the distance between the predicted marker position and its closest detected marker position is greater than a threshold distance 1312, no further updates to the position may be performed 1318. If the distance between the predicted marker position and its closest detected marker position is less than the threshold distance 1312, adjacent or nearby positions may be cleared 1314. A filter may be updated 1316 based on the cleared neighborhood. The marker position may then be determined 1318.

[0212] Fig.14 Results of marker detection and tracking are illustrated. Frame 1402 illustrates a real-time low-dose XA, and frame 1404 illustrates a selected high-dose XA. As described above and herein, the high-dose XA can be used to generate a vessel map and / or identify a working vessel. The real-time low-dose XA and high-dose XA can be correlated using the methods described above and herein to provide dynamic visualization of the device during delivery of the device within the vessel. A marker 1408 on the device can be identified in real-time in either or both of the low-dose XA 1402 and the high-dose XA 1404. According to some examples, an indication of the marker 1408 can be provided for output in one or both of the low-dose XA 1402 and the high-dose XA 1404.

[0213] According to some examples, an output bounding box 1406 may be provided. The bounding box 1406 may be sized based on the type of device. The bounding box 1406 may define the area of ​​the device. The area of ​​the device may be determined based on the detected marker positions. The markers 1408 detected in the real-time low-dose XA 1402 may be superimposed on the high-dose XA 1404 to provide a dynamic visualization of the location of the device within the working vessel.

[0214] Exemplary Methods

[0215] Fig.17 An exemplary flow chart for dynamically visualizing the delivery of a device within a blood vessel is illustrated. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0216] Workflow 1700 can be used, for example, as part of extraluminal image driven PCI. The extraluminal image can be an angiographic image. Workflow 1700 can be initiated by obtaining a pre-contrast high-dose extraluminal image 1702 and a post-contrast high-dose extraluminal image 1704. The pre-contrast high-dose extraluminal image can be, for example, an extraluminal image obtained or captured before a contrast agent is injected into a blood vessel. The post-contrast high-dose extraluminal image can be, for example, an extraluminal image obtained or captured after a contrast agent is injected into a blood vessel. The post-contrast high-dose extraluminal image can be captured while the contrast agent is in the imaged blood vessel.

[0217] According to some examples, workflow 1700 can include obtaining intravascular data 1722. Intravascular data 1722 can be obtained using an intravascular device (e.g., an OCT probe, an IVUS catheter, a micro-OCT probe, a NIRS sensor, an OFDI, a pressure line, a flow meter, or the like). Intravascular data 1722 can include, for example, intravascular images, pressure measurements, flow measurements, and the like.

[0218] The intravascular data 1722 may be co-registered 1724 with an extraluminal image (eg, the pre-contrast high-dose extraluminal image 1702 and / or the post-contrast high-dose extraluminal image 1704 ).

[0219] According to some examples, if the intravascular data 1722 is not co-registered with the extraluminal image, a landing zone can be determined 1726. According to some examples, the landing zone 1726 can be determined by a user (e.g., a physician). In some examples, an occlusion or other disease-related feature can be identified in a post-contrast angiogram. In some examples, the occlusion or disease-related feature can be automatically identified based on vascular data captured by the intraluminal device and / or the extraluminal device. For example, a restricted contrast flow in a post-contrast extraluminal image can be used to identify an occlusion or disease-related feature. In some examples, the system can receive input corresponding to the identification of an occlusion or disease-related feature via a GUI. The identified occlusion or other disease-related feature can be when determining a landing zone.

[0220] The co-registered extraluminal image and intravascular data 1724 and / or landing zone determination 1726 can be used to determine information 1728 related to lesions, treatments, and the like. Lesion-related information can include, for example, calcified area, lipid area, plaque length, plaque location, percent stenosis, and the like. Treatment-related information can include, for example, identifying one or more candidate landing zones for a stent, balloon, vessel preparation device, and the like. In some examples, information 1728 can include plaque burden, fractional flow reserve ("FFR") measurements at one or more locations along a vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based metrics, stent decisions, scores, recommendations for debulking, recommendations for subsequent surgery, or the like.

[0221] The deep learning model 1706 can be applied to the post-contrast high-dose extraluminal images to obtain the vascular tree morphology. The deep learning model can be, for example, an SVS-Net convolutional neural network. The deep learning model can be used to segment the vascular tree from the high-dose extraluminal images. The vascular tree can include, for example, working vessels, side branches, and any other similar vessels. The deep learning model 1706 can be trained using annotated extraluminal images. In some examples, the annotation can be automatically completed by using another pre-trained neural network. The pre-trained neural network can be, for example, the same SVS-Net neural network configured for different software or hardware.

[0222] The deep learning model 1706 can be trained to predict a probability map that measures each pixel's membership to a vascular tree category. The membership can be, for example, corresponding to whether a location on the extraluminal image is from a vascular tree. For example, the deep learning model can predict the probability that each location on the high-dose extraluminal image is from a vascular tree. According to some examples, a larger probability value indicates that the location on the high-dose extraluminal image is more likely to be from a vascular tree. Conversely, a lower probability value can indicate that the location on the high-dose extraluminal image is less likely to be from a vascular tree. The scale of the probability values ​​can be, for example, zero to one (0-1). In such an example, a larger probability value can be a value closer to one (1), while a lower probability value can be a value closer to zero (0).

[0223] According to some examples, the scale may be 0-100, 12-22, etc. Additionally or alternatively, a lower probability value may indicate that the location is more likely to be from a vessel tree, and a higher probability value may indicate that the location is less likely to be from a vessel tree. Accordingly, a scale of 0-1 with a higher probability indicating that the location is more likely and a lower probability value indicating that the location is less likely is only an example and is not intended to be limiting.

[0224] The vessel tree can be filtered. For example, the vessel tree can be filtered by prior knowledge of vessel morphology. Vessel morphology can be detected from a pre-contrast extraluminal image and propagated to a subsequent post-contrast extraluminal image. In some examples, a deep learning model (e.g., MaskTrack+Vnet model 310) can be used to propagate vessel information.

[0225] The output of the deep learning model 1706 can be, for example, a segmented vascular tree. The segmented vascular tree can exclude regions from a guide catheter. In some examples, the segmented vascular tree can include regions from an intraluminal imaging catheter. Regions from an intraluminal imaging catheter can be detected from a pre-contrast extraluminal image. A region can include, for example, a portion of an image frame that includes an imaging catheter but not a guide catheter. According to some examples, the output of the deep learning model 1706 (e.g., a segmented vascular tree) can be used for dynamic visualization.

[0226] A cardiac cycle can be determined 1708. A high dose extraluminal sequence acquired prior to insertion of a device and / or prior to PCI can be cached for rapid access during a treatment phase. A treatment phase can include, for example, delivery of a device (e.g., a stent, balloon, or the like) in a vessel. The sequence of extraluminal images can cover at least one cardiac cycle. A cardiac cycle length can be determined using a heartbeat estimate. Based on Fig.11 Describe an example of heartbeat estimation.

[0227] According to some examples, heartbeat estimation can use a multi-scale feature library (e.g., multi-scale motion features 1102). The multi-scale feature library can include motion features at the working vessel level 702, the cable tip level 704, the vessel pixel level 706, or any other level representing the heartbeat. In some examples, the motion features can be absolute and relative motion of the working vessel, the guide catheter tip, the distal cable tip, an intermediate point along the working vessel, or the like. The absolute motion features and the relative motion features can correspond to, for example, coordinates and optical flows of points in a high-dose extraluminal image and / or a low-dose extraluminal image.

[0228] ACF (e.g. Fig.11 The ACF 1104 described in 1104) can be applied to multi-scale features to estimate heartbeats. In some examples, peaks and troughs of the ACF from the moving features can be identified. The separation in the peaks and troughs can be quantized (e.g., by filter 1110) and filtered. According to some examples, NMS (e.g., NMS 1108) can be used to implement filtering of peaks and troughs. Additional or alternative filters (e.g., median filters or other statistical filters) can be used to identify heartbeat cycles.

[0229] As part of workflow 1700, a low-dose extraluminal image 1710 may be obtained. Low-dose extraluminal image 1710 may be captured during delivery of an intravascular device. Low-dose extraluminal image 1710 may be, for example, a fluoroscopic image. According to some examples, low-dose extraluminal image 1710 may substantially correspond to Figure 3A Frame 324 and / or Fig.12 Frame 1202 in.

[0230] The low-dose extraluminal image 1710 can be used to detect device markers 1712. The device can be, for example, a balloon device, an imaging probe, a pressure wire, a stent delivery device, a vessel preparation device (e.g., an atherectomy device or lithotripsy device, or the like). The device can include radiopaque markers or markers that are visible in the extraluminal image.

[0231] Artificial intelligence (e.g., AI models, convolutional neural networks, etc.) can be used to detect 1712 markers on the device. For example, a V-Net convolutional neural network, a MaskTrack+Vnet model, etc. can be used to detect marker candidates from the same low-dose extraluminal image. According to some examples, a V-Net marker detector can be trained based on annotated marker locations of intraluminal devices (e.g., intraluminal imaging probes or balloon delivery devices) in extraluminal images. High-dose extraluminal images and low-dose extraluminal images in the training data can be separated to employ a transfer learning strategy.

[0232] A marker detector (e.g., a V-Net model) can be trained to predict a probability map. According to some examples, a marker detector can be trained to predict the probability that a location on a low-dose extraluminal image is a marker. In some examples, the probability map can be thresholded to segment the device marker. Thresholding can include, for example, segmenting pixels inside the device marker based on a probability map of the image. For example, the marker detector can predict a probability map of each pixel being inside the device marker. In some examples, the coordinates of the device marker center point can be determined based on the thresholded probability map during post-processing of the image. For example, the device marker center coordinates can be determined by identifying connected regions and detecting the center point of each region. The marker position can be propagated from a previous frame using a motion estimate of the marker. In some examples, the motion can be determined from optical flow. According to some examples, the propagated device marker position from the previous frame can be updated by the marker position detected by the neural network in the current frame. For example, a linear Kalman filter can be used to update the device marker position. Details on how to detect and track the marker will be described in detail ... Fig.12 and Fig.13 Discuss in more detail.

[0233] According to some examples, the detected markers can be enhanced 1712. For example, the detected markers can be enhanced to increase visualization of the markers on the low-dose extraluminal image and / or the high-dose extraluminal image. Enhancing the markers can increase visualization of the markers by distinguishing the markers in terms of contrast, color, etc. compared to surrounding image features. This can make it easier for a user to identify the markers and focus on a given area of ​​the high-dose extraluminal image and / or the low-dose extraluminal image provided for output as part of the dynamic visualization.

[0234] Local image processing can be used to enhance the detected markers. In some examples, local image processing can be implemented by local contrast stretching. Various intensity transform-based methods can be applied to stretch the local image contrast.

[0235] In the example where intravascular data 1722 is obtained and information 1728 is determined, the information 1728 , the heartbeat cycle 1708 , and the detected device markers 1712 may be phase matched to provide a dynamic visualization 1714 .

[0236] Phase matching can include, for example, matching the phase of high-dose extraluminal images 1702, 1704 and low-dose extraluminal image 1710. Phase matching can be accomplished using motion features. For example, the high-dose extraluminal images and the low-dose extraluminal images can be phase matched in time without using an EKG or other physiological signal or timestamp that measures the patient's heartbeat or breathing pattern. Instead, the temporal phase matching can be based on image motion features. According to some examples, the high-dose extraluminal images and the low-dose extraluminal images can be phase matched in space. Details on how to phase match the high-dose extraluminal images and the low-dose extraluminal images will be referred to in detail. Figure 8 and Fig. 9 Discuss in more detail.

[0237] Dynamic visualization may include, for example, providing a roadmap image relative to a low-dose extraluminal image as an output. The roadmap image may be, for example, a vascular map. In some examples, detected markers may be provided for output on the vascular map and / or the low-dose extraluminal image to provide an indication of the position of the device in real time without having to inject additional contrast agent. According to some examples, information 1728 may be provided for output on or relative to the vascular map and / or the low-dose extraluminal image. For example, a treatment area initially identified on the vascular map may be provided for display on the low-dose extraluminal image so that the position of the device on the low-dose extraluminal image relative to the treatment area can be seen. In some examples, markers may be provided for output on the vascular map to provide an indication of the position of the device relative to the treatment area in real time.

[0238] According to some examples, the temporal phase matching may have to be adjusted. Adjusting the phase matching may include, for example, adjusting, selecting another high-dose extraluminal image with a low-dose extraluminal image, or matching another high-dose extraluminal image with a low-dose extraluminal image. Adjusting the temporal synchronization 1716 may take into account errors that may occur during automatic temporal phase matching. For example, if the patient has an irregular heartbeat, enters an irregular rhythm during surgery, or the like, adjustments for temporal synchronization 1716 may be used to account for these differences.

[0239] Fig.18 An example of adjusting the temporal synchronization of a high-dose extraluminal image and a low-dose extraluminal image is illustrated. Frame 1802 illustrates the segmented vessel tree provided by the deep learning model 1706 as an overlay on the low-dose extraluminal image 1710. The segmented vessel tree in frame 1802 is automatically temporally matched 1714 with the low-dose extraluminal image.

[0240] In frame 1804, the time synchronization of the vessel tree and the low-dose extraluminal image is adjusted. For example, the selected vessel tree may be two frames earlier than the vessel tree in phase matching 1714. In frame 1806, the selected vessel tree may be five frames earlier than the vessel tree in phase matching 1714.

[0241] According to some examples, the adjustment can be completed by a user (e.g., a physician). The physician can select a vascular tree from a cardiac cycle, a specific point in a cardiac cycle, and the like. In some examples, the vascular tree can be selected as a static image so that the vascular tree provided for output does not change regardless of that portion of the cardiac cycle. In such an example, a low-dose extraluminal image (e.g., a live fluoroscopic image) can be aligned with the static vascular tree during at least a portion of the cardiac cycle. In some examples, adjustment 1716 can allow the static vascular tree to be adjusted or changed. According to some examples, adjustment 1716 can be completed using various user control interfaces (e.g., a knob control, a keyboard, a joystick, a touch screen, or the like).

[0242] Fig.19 and Fig. 20 Another example of adjusting the time synchronization of the high-dose extraluminal image and the low-dose extraluminal image is illustrated. Fig.19 and Fig. 20 Three frames of low-dose extraluminal images in different cardiac phases of the same cycle are illustrated, along with corresponding dynamic visualization elements with phase matching 1714 and default adjustments 1716 after phase matching. As shown in the image frames, the high-dose extraluminal image sequence and the low-dose extraluminal image sequence exhibit significant differences, which may be due to rewiring. In such an example, the initial phase matching 1714 may not adequately synchronize the cardiac phases in the high-dose extraluminal image sequence and the low-dose extraluminal image sequence. Adjusting the initial high-dose extraluminal image of the sequence using subsequent adjustments 1716 of the high-dose extraluminal image can more accurately synchronize the high-dose extraluminal image sequence and the low-dose extraluminal image sequence.

[0243] refer to Fig.19As examples, image frame 1902 illustrates a low-dose extraluminal frame (e.g., frame 15) of the sequence of frames, image frame 1904 illustrates another low-dose extraluminal frame (e.g., frame 18) of the sequence of frames, and image frame 1906 illustrates yet another low-dose extraluminal frame (e.g., frame 21) of the sequence of frames. Image frame 1912 illustrates phase matching frame 15 to a high-dose extraluminal frame using phase matching 1714, image frame 1914 illustrates phase matching frame 18 to a high-dose extraluminal frame using phase matching 1714, and frame 1916 illustrates phase matching frame 21 to a high-dose extraluminal frame using phase matching 1714. Image frame 1922 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 15, image frame 1924 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 18, and image frame 1926 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 21. Image frame 1932 illustrates an overlay of a device marker detected from frame 15 overlaid on the high-dose extraluminal image, image frame 1934 illustrates an overlay of a device marker detected from frame 18 overlaid on the high-dose extraluminal image, and image frame 1936 illustrates an overlay of a device marker detected from frame 21 overlaid on the high-dose extraluminal image. In this example, the alignment of visual landmarks between the high-dose extraluminal images 1702, 1704 and the low-dose extraluminal image 1710 using phase matching 1714 results in poor temporal synchronization.

[0244] To improve Fig.19 For example time synchronization, adjustment 1716 may be performed. Fig. 20 An example of how adjustment 1716 can improve time synchronization is illustrated. For example, a frame of a low-dose extraluminal frame can be matched with a high-dose extraluminal image at substantially the same cardiac phase. In some examples, substantially the same cardiac phase can correspond to the same cardiac motion. The low-dose extraluminal image can be overlaid on the automatically phase-matched high-dose image, such as Fig.19 However, instead of accepting the automatic phase matching overlay, for the low-dose extraluminal image in the live feed, a second high-dose extraluminal image can be selected and overlaid on the low-dose extraluminal image, as shown in FIG. Fig. 20 In some examples, an additional offset may be added to the frame index of each automatically phase matched high dose extraluminal image frame to specify the frame index of the second high dose extraluminal image for overlay. The offset may be cached and updated. The offset may be updated automatically and / or via user input.

[0245] exist Fig. 20, image frame 2002 illustrates a low dose extraluminal frame (e.g., frame 15) of the sequence of frames, image frame 2004 illustrates another low dose extraluminal frame (e.g., frame 18) of the sequence of frames, and image frame 2006 illustrates yet another low dose extraluminal frame (e.g., frame 21) of the sequence of frames. Image frame 2012 illustrates frame 15 being phase matched to a high dose extraluminal frame using phase matching 1714 and then adjusted 1716, image frame 2014 illustrates frame 18 being phase matched to a high dose extraluminal frame using phase matching 1714 and then adjusted 1716, and frame 2016 illustrates frame 21 being phase matched to a high dose extraluminal frame using phase matching 1714 and then adjusted 1716. Image frame 2022 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 15, image frame 2024 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 18, and image frame 2026 illustrates an overlay of a vessel tree segmented from a high-dose extraluminal image overlaid on frame 21. Image frame 2032 illustrates an overlay of a device marker detected from frame 15 overlaid on the high-dose extraluminal image, image frame 2034 illustrates an overlay of a device marker detected from frame 18 overlaid on the high-dose extraluminal image, and image frame 2036 illustrates an overlay of a device marker detected from frame 21 overlaid on the high-dose extraluminal image.

[0246] Back to Fig.17 , after adjusting the time synchronization 1716, the dynamic visualization can be updated 1718 for output. The dynamic visualization can provide an output including any of a vascular map, a live tracking of the position of the device, an indication of the treatment zone, an indication of the position of the device relative to the treatment zone, or the like. In some examples, the output may include multiple images, videos, video screenshots, or the like. For example, two or more images and / or videos may be output relative to each other. One image may be a vascular map, and another image may be a live image tracking the position of the device. In some examples, the output may allow a physician to dynamically visualize the delivery of the device within the blood vessel using less contrast agent. According to some examples, for example, when the intravascular data 1722 is obtained, the intravascular data may be co-registered with respect to the vascular map and / or the low-dose extraluminal image and provided for output. In some examples, one or more additional representations of the intravascular data 1722 may be provided for output (e.g., a map, an axial representation of a vessel, or the like). The map may be, for example, a graphical representation of FFR, EEL, pressure measurement, or the like. The axial representation may be a symmetrical representation of a vessel. Axial representation may be symmetric about the longest axis (eg, the longitudinal axis).

[0247] According to some examples, the output can be screenshot and / or scaled 1720. When the device reaches the area of ​​interest, the output can be automatically screenshot. For example, the detected markers 1712 can be used to determine the position of the device relative to the area of ​​interest. The area of ​​interest can be, for example, a treatment device landing zone, a balloon device zone, a vessel preparation device zone, or a lesion-related zone. In some examples, the area of ​​interest can be a portion of a selected blood vessel, a blood vessel line, or the like.

[0248] When the detected marker is within a threshold distance of the region of interest, the system can automatically take a screenshot of the output. The threshold distance can be determined based on the number of pixels between the outer boundary of the region of interest and the detected marker of the device. In some examples, the threshold distance can be determined based on the spatial distance (e.g., Euclidean distance or geodesic distance) between the detected marker and the region of interest.

[0249] According to some examples, a user (e.g., a doctor) can provide input to take a screenshot of the output. A screenshot can be provided for the output. According to some examples, a user can confirm the position of the device based on the screenshot without having to inject additional contrast agent.

[0250] The dynamic visualization and / or screenshot can be manually or automatically enlarged to display a high-resolution view of an area of ​​interest. In some examples, the area of ​​interest can correspond to the location of the device, a treatment area, or the like. According to some examples, the output of the dynamic visualization and / or the output of the screenshot can be automatically enlarged on the location of the device. In one example, as the device approaches the treatment area, the output of the dynamic visualization can be automatically enlarged to focus on the device. The spatial distance between the detected marker and the treatment area can be used to determine whether the device is approaching the treatment area and / or has reached the treatment area. In some examples, the number of pixels between the outer boundary of the treatment area and the detected marker can be used to determine whether the device is approaching the treatment area and / or has reached the treatment area.

[0251] In some examples, local contrast stretching can be used to enhance selected portions (e.g., regions of interest) of the dynamic visualization output and / or screenshot output. The local contrast between the device marker and the background can be stretched within the detected and tracked region of interest. By selectively enhancing the local contrast between the marker and the surrounding area, the model conserves computational resources compared to enhancing the entire working vessel because the remaining computational resources are consumed.

[0252] According to some examples, the pixel values ​​within the region of interest may be filled and sorted into an approximate representation (e.g., a data graph or histogram). In some examples, a threshold t for intensity may be set by a user or by an AI model. According to some examples, the threshold t may be a pixel value corresponding to the 1.5th percentile in the data representation. The selection of the 1.5th percentile as the threshold t is not fixed. Instead, the threshold t may vary to ensure that it remains above the device-marked pixel values. In some examples, pixels with intensity values ​​below the threshold t may be normalized to within the interval [0, t]. A median filter may be applied to the region of interest where the intensity is stretched to smooth out noise exhibiting pixel values ​​similar to the device-marked pixel values. According to some examples, the filter size may be set to 3×3.

[0253] To ensure consistency in background pixel value distribution inside and outside a given region of interest, local contrast stretching can be further localized near device markers. According to some examples, for each region of interest, pixel values ​​before and after local contrast stretching can be cached, and the difference can be applied in a spatially dependent manner as follows:

[0254] I=(I′-I0)×W+I0,

[0255] Where I0 and I' are pixel values ​​from the same region of interest before and after local contrast stretching, and W is a spatial weight function. According to some examples, W can be a Gaussian kernel centered about a center point of a device marker with respect to the region of interest. In the case where there are multiple markers within the region of interest, W can be a mixture of each Gaussian kernel centered about a center point of one marker. According to some examples, W can be normalized to [0,1] to selectively apply pixel-level intensity variations within the region of interest.

[0256] FIG. 27A to FIG. 27CA comparison between conventional zooming and local contrast stretching of an area of ​​interest is illustrated. Frames 2741, 2742, and 2743 are complete XA images showing the full field of view of a working vessel. In each of frames 2741, 2742, and 2743, the area of ​​interest is bounded by a box. In these examples, each area of ​​interest contains two device markers. These frames are low-dose XAs acquired in a manner similar to images captured in a clinical setting. Each frame shows a different type of working vessel. Frames 2751, 2752, and 2753 are enlarged images of the corresponding area of ​​interest without enhancement. Frames 2761, 2762, and 2763 are enhanced stretched images of the corresponding area of ​​interest. Enhanced frames 2761, 2762, and 2763 are enlarged images of the area of ​​interest enhanced using the above method. In enhanced frames 2761, 2762, and 2763, the device markers are more clearly visible because they appear darker and more distinct than the corresponding frames 2751, 2752, and 2753.

[0257] Fig.21 An exemplary real-time state transition is illustrated. Motion stabilization or motion compensation may be used for online matching 2100. In some examples, online matching 2100 may correspond to Fig.17 Phase matching 1714 described. According to some examples, online matching 2100 can correspond to phase matching used during a live feed of a low-dose extraluminal image sequence. Losses can be determined and used to evaluate errors in the matching process. Losses can be determined based on a distance transform from spatial landmarks in the low-dose extraluminal image to a working vessel map in a motion-stabilized and motion-compensated high-dose extraluminal image. Losses can be compared to a threshold "T". In some examples, if the losses are less than the threshold, dynamic visualization can be provided for the output, while if the losses are greater than the threshold, additional corrections 2104 may be required before dynamic visualization is provided for the output. Corrections 2104 can include, for example, adjustments 1716, such as regarding Fig.17 , Fig. 20 and Fig.21 As described herein. Correction 2104 may be initiated automatically and / or by a user. For example, user-initiated correction 2104 may occur based on visual feedback not being captured by a determined loss. In some examples, visual feedback for correction 2104 may be based on the proximity of a spatial landmark in a low-dose extraluminal image to a working vessel map from a motion-stabilized and motion-compensated high-dose image. The spatial landmark may be, for example, a guide catheter tip, a cable tip, a balloon body, etc.

[0258] The high-dose extraluminal images in the selected sequence may be phase matched to the low-dose extraluminal images based on their temporal order using phase matching 1714 and / or adjustment 1716. When the selected high-dose extraluminal image is used for phase matching once, the system may loop back to the first high-dose extraluminal image to reset phase matching 2106. Phase matching of incoming low-dose extraluminal images (e.g., live fluoroscopic images) may be repeated on the same sequence of high-dose extraluminal images.

[0259] According to some examples, the system can perform a treatment zone check 2108. The treatment zone check 2108 can determine whether the detected device marker 1712 is close to a predefined treatment zone (e.g., a stent landing zone or a balloon landing zone). In some examples, the treatment zone check 2108 can be based on a spatial distance, such as a Euclidean distance, a geodesic distance, a pixel distance, or the like. If the determined distance is within a threshold distance of the treatment zone, an output can be provided. In some examples, the output can be a notification or an alarm. The notification can be a visual, audible, or tactile notification. The notification can indicate the proximity of the device to the treatment zone.

[0260] According to some examples, the system can detect when the low-dose extraluminal image is not updated. The low-dose extraluminal image may not be updated due to an interruption in image acquisition. In an example where the low-dose extraluminal image is not updated, the system can enter a replay mode 2110. The replay mode 2110 can include a replay period. The replay period can be, for example, the number of frames between the current low-dose extraluminal image frame and the closest previous low-dose extraluminal image frame it repeats. The closest previous low-dose extraluminal image can be, for example, the most recent previously captured low-dose extraluminal image frame. In some examples, the closest previous low-dose extraluminal image can be the closest low-dose extraluminal image frame based on distance (e.g., based on image capture location). According to some examples, dynamic visualization elements (e.g., treatment area, device location, etc.) can be provided for output during the replay mode 2110.

[0261] The system can compare the current low-dose extraluminal image frame to the previous low-dose extraluminal image frame in the given sequence. When the current low-dose extraluminal image frame corresponds to at least one of the low-dose extraluminal image frames in the sequence, the system can remain in the replay mode 2110. When the low-dose extraluminal image frame is different from the previous low-dose extraluminal frame in the sequence, the system can exit the replay mode 2110. In such an example, once the system exits the replay mode 2110, the system can return to the online matching 2100.

[0262] Fig.16An exemplary method for dynamically visualizing the delivery of a device within a blood vessel is illustrated. The following operations do not have to be performed in the exact order described below. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0263] At least one first extraluminal image may be received in block 1610. The at least one first extraluminal image may be, for example, a high-dose contrast agent XA. According to some examples, a vessel map may be generated based on the first extraluminal image.

[0264] According to some examples, the vessel map may include an indication of a working vessel. The working vessel may be automatically detected using an AI trained to automatically detect the working vessel. Training the AI ​​may include annotating a pre-contrast extraluminal image as a line that follows the trajectory of at least one of a guide wire or a guide catheter, marking the path of the working vessel in a post-contrast extraluminal image, providing the annotated pre-contrast extraluminal image and the annotated post-contrast extraluminal image as training data to the AI ​​model, and training the AI ​​model to predict the vessel trajectory. According to some examples, the AI ​​may include an AI model trained to automatically detect the working vessel.

[0265] The AI ​​model can be executed to predict or identify a working vessel trajectory. For example, at least one pre-contrast extraluminal image can be received as an input into the AI ​​model. Based on the at least one pre-contrast extraluminal image, the AI ​​model can detect a wire of an intravascular device. Based on the detected wire, the AI ​​model can propagate cable information. The AI ​​model can automatically predict a working vessel trajectory based on the propagated cable information.

[0266] In block 1620, a second extraluminal image captured during delivery of an intravascular device may be received. The second extraluminal image may be, for example, a low-dose contrast agent XA. The intravascular device may be, for example, a stent delivery device, a balloon device, an intravascular imaging probe, or a pressure wire.

[0267] The at least one first extraluminal image may be captured or generated at a different time than the second extraluminal image. For example, the at least one first extraluminal image may be generated during a diagnostic phase, a surgical planning phase, or the like, while the second extraluminal image may be captured during a procedure (e.g., delivery of a device within a blood vessel).

[0268] In block 1630, a moving feature in the at least one first extraluminal image and the second extraluminal image may be detected. The moving feature may include at least one of a guide catheter tip, a distal endpoint of a working vessel, an optical flow at the guide catheter tip, or an optical flow at a distal endpoint of the working vessel. According to some examples, detecting the moving feature may further include automatically detecting the working vessel by executing an AI model. In some examples, a heartbeat cycle of the patient may be determined based on the detected moving feature.

[0269] In block 1640, the at least one first extraluminal image and the second extraluminal image may be correlated based on the detected movement feature. Correlating the at least one first extraluminal image and the second extraluminal image may include determining vessel level movement, cable tip level movement, and vessel pixel level movement. Determining vessel level movement may include determining a two-dimensional translation vector. The two-dimensional translation vector may correspond to a two-dimensional translation at the nth frame relative to the first frame. Determining cable tip level movement may include determining absolute spatial information and relative spatial information. The absolute spatial information may correspond to coordinates, and the relative spatial information may correspond to an optical flow between adjacent image frames. Determining vessel pixel level movement may include determining absolute spatial information and relative spatial information.

[0270] According to some examples, based on the detected movement features, a spatiotemporal phase match between at least one of the at least one first extraluminal image and the second extraluminal image can be determined. For example, the detected movement features can be resampled at a common frame rate. A maximum correlation coefficient of the movement feature pairs in the at least one first extraluminal image and the second extraluminal image can be determined. A time shift can be determined based on the maximum correlation coefficient. In some examples, a drift between the detected movement features in the at least one first extraluminal image can be determined. The time shift can be adjusted based on the determined drift. According to some examples, the time shift can be iteratively filtered. The time shift can be updated to a corrected time shift based on the iteratively filtered time shift.

[0271] According to some examples, the spatiotemporal phase matching can be adjusted. For example, another first extraluminal image that is different from at least one of the first extraluminal images can be identified. The spatiotemporal phase matching can be adjusted based on the another first extraluminal image. The another first extraluminal image can be, for example, another first extraluminal image from a sequence of first extraluminal images. In such an example, adjusting the spatiotemporal phase matching can include shifting the sequence of first extraluminal images based on the identified another first extraluminal image. The real-time visualization can be updated based on the another first extraluminal image.

[0272] In block 1650, a real-time visualization of the location of the intravascular device may be provided for output on one of the at least one first extraluminal image or the second extraluminal image that includes the intravascular device.

[0273] According to some examples, an intravascular device can be detected in a second extraluminal image. Detecting the intravascular device can include, for example, detecting one or more markers of the intravascular device. The intravascular device and / or the markers of the intravascular device can be detected by executing an AI model. The AI ​​model can be trained by providing a co-registered dataset including a plurality of intraluminal images and extraluminal images as input. The intraluminal image and the extraluminal image can be annotated images. The annotated image can include annotations identifying one or more intravascular device markers. The AI ​​model can be trained to predict the location of the intravascular device.

[0274] According to some examples, detecting the intravascular device may include detecting an optical flow of the intravascular device. A position of the intravascular device in a first frame of the second intraluminal image may be determined based on the optical flow. A position of the intravascular device in a subsequent frame may be determined based on the optical flow. In some examples, the position of the intravascular device in the subsequent frame may be further based on the position of the intravascular device in the first frame.

[0275] In some examples, detecting an intravascular device may include detecting an optical flow of the intravascular device. The position of the intravascular device in a first frame of the second extraluminal image may be determined by executing an AI model based on at least a first frame of the second extraluminal image. The position of the intravascular device in a subsequent frame of the extraluminal image may be predicted based on the optical flow detected in a previous frame of the second extraluminal image and the position of the intravascular device. The position of the intravascular device in a subsequent frame of the second extraluminal image may be updated by executing an AI model based on at least one subsequent frame of the second extraluminal image and the predicted position of the intravascular device in the subsequent frame of the second extraluminal image.

[0276] In some examples, the output may include an indication of a treatment zone on at least one of the second extraluminal images or on the at least one first extraluminal image. The treatment zone may be, for example, a stent landing zone or a balloon landing zone.

[0277] According to some examples, the system can provide a notification as an output. A notification can be provided when an intravascular device is inserted before, near, within, or through a treatment region.

[0278] According to some examples, a screenshot of a real-time visualization of the location of an intravascular device can be captured. In some examples, a screenshot can be automatically captured when the intravascular device is within a threshold distance of an area of ​​interest. The area of ​​interest can be a treatment zone. The treatment zone can be at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone. According to some examples, the area of ​​interest can be automatically determined based on intravascular data captured by one or more extraluminal devices and / or intraluminal devices. For example, a treatment zone (e.g., a stent landing zone) can be automatically determined based on one or more EEL values, calcium values, plaque loads, vascular diameter values, or the like. In some examples, the area of ​​interest can be received as an input. For example, the system can receive an input corresponding to a selection of an area of ​​interest. The input can be a position on or along a blood vessel selected by, for example, a user (such as a doctor) via a user interface.

[0279] Determining the threshold distance may include at least one of determining the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or determining a spatial distance between the at least one detected marker on the intravascular device and the region of interest. The spatial distance may be a Euclidean distance, a geodesic distance, etc.

[0280] According to some examples, a zoom feature can be implemented to magnify a portion of a real-time visualization. When the intravascular device is within a threshold distance of a region of interest, the portion of the real-time visualization can be automatically zoomed. The portion of the real-time visualization can correspond to the region of interest. The region of interest can be, for example, a treatment zone or the location of the intravascular device. The treatment zone can be at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone. Determining the threshold distance can include at least one of determining a number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or determining a spatial distance between the at least one detected marker on the intravascular device and the region of interest. The spatial distance can be a Euclidean distance or a geodesic distance.

[0281] The systems and methods described above and herein provide treatment visualization and PCI navigation (e.g., dynamic visualization) stabilized by movement. Dynamic visualization can be implemented at different magnification ratios, allowing the physician to view as much or as little detail as desired. In addition, dynamic visualization can also be implemented at different instances during surgery. In some examples, the system can receive an input corresponding to a request for dynamic visualization for a threshold time period (e.g., a few seconds). In some examples, dynamic visualization can be used as a substitute for injecting additional contrast agents in sequence. For example, dynamic visualization can correspond to virtual smoke (puff) that simulates and therefore replaces the smoke injected with a small amount of contrast agent to visualize the blood vessel. In this virtual smoke scenario, the system will include at least some (if not all) of the processing steps described above and herein to detect landmarks in low-dose XA, calculate movement features, and perform phase matching. Then display the blood vessels, lesions, devices, landing areas, and related information from the phase-matched high-dose XA in the low-dose XA view. The system can activate the virtual smoke in response to the input. For example, the system may include an input (e.g., a button, a computer mouse, a joystick, or a virtual syringe) activated by a physician. In some examples, virtual smoke may also be triggered by following a standard smoke step / device but without injecting or using a real contrast agent. The period of visualization may be estimated based on a lookup table approach. For example, if a predetermined amount of contrast agent (e.g., X ml of contrast agent) can last for a threshold time period (e.g., Y seconds), the visualization system may be configured to provide dynamic information for the threshold time period (e.g., Y seconds) and then fade out. Further, the output provided by dynamic visualization allows detection, tracking of intravascular devices, and projection of intravascular devices onto a vascular map. This enables physicians to clearly visualize the vascular morphology while delivering intravascular devices and only taking low-dose XA shots. In addition, dynamic visualization allows for effective and almost completely automatic fusion of low-dose XA and high-dose XA information.

[0282] The various aspects, features and examples of the present disclosure should be considered to be illustrative in all aspects and are not intended to limit the present disclosure, the scope of which is limited only by the claims. For those skilled in the art, other examples, modifications and uses will be apparent without departing from the spirit and scope of the present invention.

[0283] Throughout the application, when a composition is described as having, including, or comprising specific components, or when a method is described as having, including, or comprising specific method steps, it can be considered that the composition of the present teachings also consists essentially of (or consists of) the described components, and the method of the present teachings also consists essentially of (or consists of) the described method steps.

[0284] In this application, when an element or component is said to be included in and / or selected from a list of recorded elements or components, it should be understood that the element or component can be any one of the recorded elements or components and can be selected from a group consisting of two or more recorded elements or components. In addition, it should be understood that the elements and / or features of the compositions, devices or methods described herein can be combined in various ways without departing from the spirit and scope of the invention, whether explicitly or implicitly stated herein.

[0285] The use of the terms "including," "comprising," or "having" should be generally interpreted as open ended and non-limiting unless specifically stated otherwise.

[0286] Unless otherwise specifically stated, singular forms used herein include plural forms (and vice versa). In addition, the singular forms "a", "an" and "the" include plural forms unless the context clearly dictates otherwise. In addition, when the term "approximately" is used before a quantitative value, unless otherwise specifically stated, the present teachings also include the specific quantitative value itself. As used herein, the term "approximately" refers to a variation of ±10% of the nominal value. All numerical values ​​and ranges disclosed herein are deemed to include "approximately" before each numerical value.

[0287] It should be understood that the order of steps or the order in which certain actions are performed are not important as long as the present invention remains operable. In addition, two or more steps or actions may be performed simultaneously.

[0288] Where ranges or lists of values ​​are provided, each interval between the upper and lower limits of the ranges or lists of values ​​is contemplated individually and is included in the present invention as if each value were specifically listed herein. In addition, smaller ranges between and including the upper and lower limits of a given range are also contemplated and included in the present invention. Listing exemplary values ​​or ranges does not mean that other values ​​or ranges between and including the upper and lower limits of a given range are abandoned.

Claims

1. A method comprising: receiving, by one or more processors, at least one first extraluminal image; receiving, by the one or more processors, a second extraluminal image captured during delivery of the intravascular device; detecting, by the one or more processors, movement features in the at least one first extraluminal image and the second extraluminal image; correlating, by the one or more processors, the at least one first extraluminal image and the second extraluminal image based on the detected motion characteristic; as well as A real-time visualization of a location of the intravascular device on one of the at least one first extraluminal image or the second extraluminal image including the intravascular device is provided for output by the one or more processors. 2 . The method of claim 1 , wherein the at least one first extraluminal image is a high-dose contrast agent X-ray angiogram.

3. The method of claim 1 or 2, wherein the second extraluminal image is a low-dose contrast X-ray angiogram.

4. The method of any of the preceding claims, wherein the intravascular device is at least one of a stent delivery device, a balloon device, an intravascular imaging probe, a vessel preparation device, or a pressure line. 5 . The method of any of the preceding claims, further comprising generating, by the one or more processors, a vessel map based on the at least one first extraluminal image.

6. The method of any of the preceding claims, further comprising automatically detecting, by the one or more processors, a working vessel using an artificial intelligence (AI) model.

7. The method of claim 6, wherein training the AI ​​model to automatically detect the working vessel comprises: annotating the pre-contrast extraluminal image as a line following a trajectory of at least one of the guide wire or the guide catheter; annotating the path of the working blood vessel in the post-contrast extraluminal image; providing the AI ​​model with the annotated pre-contrast extraluminal images and the labeled post-contrast extraluminal images as training data; and The AI ​​model is trained to predict working vessel trajectories.

8. The method according to claim 6, further comprising: receiving, by the one or more processors, at least one pre-contrast extraluminal image as input into the AI ​​model; The AI ​​model is trained by the one or more processors that enhance the high-dose extraluminal image into a low-dose extraluminal image as input to segment at least one of a guide catheter, a guidewire, a stent marker, or a balloon marker on the low-dose extraluminal image; detecting, by the one or more processors executing the AI ​​model, a guidewire of the intravascular device based on the at least one pre-contrast extraluminal image; propagating, by the one or more processors executing the AI ​​model on a frame-by-frame basis, wire information based on the detected wires; as well as A working vessel trajectory is automatically predicted by the one or more processors executing the AI ​​model based on the propagated guidewire information.

9. The method of any of the preceding claims, wherein the movement feature comprises at least one of a guide catheter tip, a distal endpoint of a working vessel, an optical flow at the guide catheter tip, or an optical flow at a distal endpoint of the working vessel.

10. The method of claim 9, wherein detecting the movement feature further comprises automatically detecting the working vessel by the one or more processors executing an artificial intelligence (AI) model.

11. The method according to any of the preceding claims, wherein correlating the at least one first extraluminal image and the second extraluminal image comprises: determining, by the one or more processors, vessel level movement; determining, by the one or more processors, cable tip level movement; as well as Vessel pixel level movement is determined by the one or more processors. 12 . The method of claim 11 , wherein determining the vessel level movement further comprises determining, by the one or more processors, a two-dimensional translation vector.

13. The method of claim 12, wherein the two-dimensional translation vector corresponds to a two-dimensional translation at the nth frame relative to the first frame.

14. The method of claim 11, wherein determining the cable tip level movement comprises determining, by the one or more processors, absolute spatial information and relative spatial information.

15. The method of claim 14, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

16. The method of claim 11, wherein determining the vessel pixel level movement comprises determining, by the one or more processors, absolute spatial information and relative spatial information for a plurality of points on the working vessel.

17. The method of claim 16, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

18. The method of any of the preceding claims, further comprising determining a heart cycle of the patient based on the detected movement characteristics.

19. The method of any of the preceding claims, further comprising determining, by the one or more processors, a spatiotemporal phase match between at least one of the at least one first extraluminal image and at least one of the second extraluminal images based on the detected movement characteristics.

20. The method according to claim 19, further comprising: resampling, by the one or more processors, the detected motion features at a common frame rate; determining, by the one or more processors, a maximum correlation coefficient for pairs of moving features in the at least one first extraluminal image and the second extraluminal image; as well as A time shift is determined, by the one or more processors, based on the maximum correlation coefficient.

21. The method according to claim 20, further comprising: determining, by the one or more processors, drift between detected motion features in the at least one first extraluminal image; as well as The time shift is adjusted, by the one or more processors, based on the determined drift.

22. The method according to claim 20, further comprising: iteratively predicting, by the one or more processors, the time shift; as well as The time shift is updated, by the one or more processors, to a corrected time shift based on the iteratively predicted time shift.

23. The method of claim 19, further comprising: The spatiotemporal phase matching is adjusted by the one or more processors.

24. The method of claim 23, wherein adjusting the spatiotemporal phase matching comprises: identifying, by the one or more processors, another first extraluminal image different from the at least one of the first extraluminal images; as well as The spatiotemporal phase matching is adjusted, by the one or more processors, based on the another first extraluminal image.

25. The method of claim 23, further comprising updating, by the one or more processors, the real-time visualization based on the adjustment to include the another first extraluminal image.

26. The method of any of the preceding claims, further comprising detecting, by the one or more processors, the intravascular device.

27. The method of claim 26, wherein detecting the intravascular device in the second extraluminal image further comprises executing an artificial intelligence (AI) model.

28. The method of claim 27, further comprising training the AI ​​model, wherein training the AI ​​model comprises: providing a co-registered dataset as input to the AI ​​model, the co-registered dataset comprising a plurality of intraluminal images and extraluminal images, wherein the plurality of intraluminal images and extraluminal images are annotated images; and The AI ​​model is trained to predict the location of the intravascular device.

29. The method of claim 28, wherein the annotated image includes annotations identifying one or more intravascular device markers.

30. The method of claim 26, further comprising: detecting, by the one or more processors, an optical flow of the intravascular device; determining, by the one or more processors, a position of the intravascular device in the first frame of the second extraluminal image based on the detected optical flow; as well as A position of the intravascular device in a subsequent frame of the second extraluminal image is predicted by the one or more processors based on the detected optical flow.

31. The method of any of the preceding claims, further comprising providing, by the one or more processors, a treatment zone on at least one of the second extraluminal images or a treatment zone on the at least one first extraluminal image for output.

32. The method of claim 31, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

33. The method of claim 32, wherein: The lesion-related region is at least one of a calcification frame, a lipid frame, or an anatomical frame, and The lesion-related region is identified by the one or more processors from another imaging modality and co-registered to the first extraluminal image.

34. The method of claim 33, wherein the another imaging modality is an intravascular imaging modality.

35. The method of any of the preceding claims, further comprising receiving, by the one or more processors, annotations of the at least one first extraluminal image or annotations of the second extraluminal image.

36. The method of claim 35, wherein when receiving an annotation, the method further comprises: receiving, by the one or more processors, one or more inputs from a user corresponding to the annotation; or Annotations are automatically determined, by the one or more processors, based on the vascular data.

37. A method according to claim 35 or 36, wherein the annotation includes one or more of the following: plaque burden, fractional flow reserve ("FFR") measurements at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent decisions, scores, debulking recommendations, subsequent surgical recommendations, stent placement areas, treatment device landing areas, balloon device areas, vessel preparation device areas, or lesion-related areas.

38. The method of any one of claims 35 to 37, further comprising updating, by the one or more processors, a second of the at least one first extraluminal image or a second of the second extraluminal images based on the received annotation.

39. The method of any of the preceding claims, further comprising automatically capturing, by the one or more processors, a screenshot of a real-time visualization of the position of the intravascular device.

40. The method of claim 39, wherein the screenshot is automatically captured when the intravascular device is within a threshold distance of a region of interest.

41. The method of claim 40, wherein the region of interest is a treatment area.

42. The method of claim 41, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

43. The method of claim 40, wherein determining the threshold distance comprises at least one of: determining, by the one or more processors, a number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined by the one or more processors.

44. The method of claim 43, wherein the spatial distance is a Euclidean distance or a geodesic distance.

45. The method of any preceding claim, further comprising automatically scaling, by the one or more processors, a portion of the real-time visualization.

46. ​​The method of claim 45, wherein the portion of the real-time visualization is automatically zoomed when the intravascular device is within a threshold distance of a region of interest.

47. The method of claim 46, wherein the portion of the real-time visualization corresponds to the region of interest.

48. The method of claim 47, wherein the region of interest is a treatment area or the location of the intravascular device.

49. The method of claim 48, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

50. The method of claim 46, wherein determining the threshold distance comprises at least one of: determining, by the one or more processors, a number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined by the one or more processors.

51. The method of claim 50, wherein the spatial distance is a Euclidean distance or a geodesic distance.

52. The method of claim 45, wherein automatically scaling comprises: sorting, by one or more processors, pixel values ​​of the real-time visualization by their intensities; Normalizing, by one or more processors, pixel intensity values ​​below a predetermined threshold; as well as A median filter is applied by one or more processors to the normalized pixel intensity values ​​for the real-time visualization.

53. A system comprising: One or more processors configured to: receiving at least one first extraluminal image; receiving a second extraluminal image captured during delivery of the intravascular device; detecting a motion feature in the at least one first extraluminal image and a motion feature in the second extraluminal image; correlating the at least one first extraluminal image and the second extraluminal image based on the detected movement characteristics; as well as A real-time visualization of a location of the intravascular device on one of the at least one first extraluminal image or the second extraluminal image including the intravascular device is provided for output.

54. The system of claim 53, wherein the at least one first extraluminal image is a high-dose contrast X-ray angiogram.

55. The system of claim 53 or 54, wherein the second extraluminal image is a low-dose contrast X-ray angiogram.

56. The system of any one of claims 53 to 55, wherein the intravascular device is at least one of a stent delivery device, a balloon device, an intravascular imaging probe, a vessel preparation device, or a pressure line.

57. The system of any one of claims 53 to 56, wherein the one or more processors are further configured to generate a vessel map based on the at least one first extraluminal image.

58. The system of any one of claims 53 to 57, wherein the one or more processors are further configured to automatically detect a working vessel using an artificial intelligence (AI) model.

59. The system of claim 58, wherein when training the AI ​​model to automatically detect the working vessel, the one or more processors are further configured to: annotating the pre-contrast extraluminal image as a line following a trajectory of at least one of the guide wire or the guide catheter; annotating the path of the working blood vessel in the post-contrast extraluminal image; providing the AI ​​model with the annotated pre-contrast extraluminal images and the labeled post-contrast extraluminal images as training data; and The AI ​​model is trained to predict working vessel trajectories.

60. The system of claim 58 or 59, wherein the one or more processors are further configured to: receiving at least one pre-contrast extraluminal image as input into the AI ​​model; The AI ​​model is trained by enhancing a high-dose extraluminal image into a low-dose extraluminal image as input to segment at least one of a guide catheter, a guidewire, a stent marker, or a balloon marker on the low-dose extraluminal image; detecting a guidewire of the intravascular device by executing the AI ​​model and based on the at least one pre-contrast extraluminal image; by executing the AI ​​model and propagating wire information on a frame-by-frame basis based on detected wires; as well as The working vessel trajectory is automatically predicted by executing the AI ​​model based on the propagated guidewire information.

61. The system of any one of claims 53 to 60, wherein the movement feature comprises at least one of: a guide catheter tip, a distal endpoint of a working vessel, an optical flow at the guide catheter tip, or an optical flow at a distal endpoint of the working vessel.

62. The system of claim 61, wherein when the movement feature is detected, the one or more processors are further configured to automatically detect the working vessel by executing an AI model.

63. The system of any one of claims 53 to 62, wherein when correlating the at least one first extraluminal image with the second extraluminal image, the one or more processors are further configured to: Determine vascular grade migration; Determine cable tip level movement; and Determine the pixel-level movement of blood vessels.

64. The system of claim 63, wherein when determining the vessel level movement, the one or more processors are further configured to determine a two-dimensional translation vector.

65. The system of claim 64, wherein the two-dimensional translation vector corresponds to a two-dimensional translation at the nth frame relative to the first frame.

66. The system of claim 63, wherein when determining the cable tip level movement, the one or more processors are further configured to determine absolute spatial information and relative spatial information.

67. The system of claim 66, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

68. The system of claim 63, wherein when determining the vessel pixel level movement, the one or more processors are further configured to determine absolute spatial information and relative spatial information for a plurality of points on the working vessel.

69. The system of claim 68, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

70. The system of any one of claims 53 to 69, wherein the one or more processors are further configured to determine a heart cycle of the patient based on the detected movement characteristics.

71. The system of any one of claims 53 to 69, wherein the one or more processors are further configured to determine a spatiotemporal phase match between at least one of the first extraluminal images and at least one of the second extraluminal images based on the detected movement characteristics.

72. The system of claim 71, wherein the one or more processors are further configured to: resampling detected motion features at a common frame rate; determining a maximum correlation coefficient for pairs of moving features in the at least one first extraluminal image and the second extraluminal image; and A time shift is determined based on the maximum correlation coefficient.

73. The system of claim 72, wherein the one or more processors are further configured to: determining a drift between detected moving features in the at least one first extraluminal image; and The time shift is adjusted based on the determined drift.

74. The system of claim 72 or 73, wherein the one or more processors are further configured to: iteratively predicting the time shift; and The time shift is updated to a corrected time shift based on the iteratively predicted time shift.

75. The system of any one of claims 72 to 74, wherein the one or more processors are further configured to adjust the spatiotemporal phase matching.

76. The system of claim 75, wherein when adjusting the spatiotemporal phase matching, the one or more processors are further configured to: identifying another first extraluminal image different from the at least one first extraluminal image of the first extraluminal images; and The spatiotemporal phase matching is adjusted based on the another first extraluminal image.

77. The system of claim 75 or 76, wherein the one or more processors are further configured to update the real-time visualization based on the adjustment to include the another first extraluminal image.

78. The system of claim 73, wherein the one or more processors are further configured to detect the intravascular device.

79. The system of claim 78, wherein the one or more processors are further configured to execute the AI ​​model when the intravascular device is detected in the second extraluminal image.

80. The system of claim 79, wherein the one or more processors are further configured to train the AI ​​model, wherein when training the AI ​​model, the one or more processors are further configured to: providing a co-registered dataset as input to the AI ​​model, the co-registered dataset comprising a plurality of intraluminal images and extraluminal images, wherein the plurality of intraluminal images and extraluminal images are annotated images; and The AI ​​model is trained to predict the location of the intravascular device.

81. The system of claim 80, wherein the annotated image includes annotations identifying one or more intravascular device markers.

82. The system of claim 78, wherein the one or more processors are further configured to: detecting an optical flow of the intravascular device; determining a position of the intravascular device in the first frame of the second extraluminal image based on the detected optical flow; and A position of the intravascular device in a subsequent frame of the second extraluminal image is predicted based on the detected optical flow.

83. The system of any one of claims 53 to 82, wherein the one or more processors are further configured to provide a treatment zone on at least one of the second extraluminal images or the at least one first extraluminal image for output.

84. The system of claim 83, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

85. The system of claim 84, wherein: The lesion-related region is at least one of a calcification frame, a lipid frame, or an anatomical frame, and The lesion-related region is identified by the one or more processors from another imaging modality and co-registered to the first extraluminal image.

86. The system of claim 85, wherein the another imaging modality is an intravascular imaging modality.

87. The system of any one of claims 53 to 86, wherein the one or more processors are further configured to receive annotations of the at least one first extraluminal image or annotations of the second extraluminal image.

88. The system of claim 87, wherein upon receiving an annotation, the one or more processors are further configured to: receiving one or more inputs from a user corresponding to the annotation; or Annotations are automatically determined based on the vessel data.

89. A system according to claim 87 or 88, wherein the annotation includes one or more of the following: plaque burden, fractional flow reserve ("FFR") measurement at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent decision, score, debulking recommendation, subsequent surgery recommendation, stent placement area, treatment device landing area, balloon device area, vessel preparation device area, or lesion-related area.

90. The system of claim 86, wherein the one or more processors are further configured to update a second one of the at least one first extraluminal image or the second extraluminal image based on the received annotation.

91. The system of any one of claims 53 to 90, wherein the one or more processors are further configured to automatically capture a screenshot of a real-time visualization of the position of the intravascular device.

92. The system of claim 91, wherein the screenshot is automatically captured when the intravascular device is within a threshold distance of an area of ​​interest.

93. The system of claim 92, wherein the region of interest is a treatment area.

94. The system of claim 93, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

95. The system of claim 92, wherein determining the threshold distance comprises at least one of: determining the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined.

96. The system of claim 95, wherein the spatial distance is a Euclidean distance or a geodesic distance.

97. The system of any one of claims 53 to 96, wherein the one or more processors are further configured to automatically scale a portion of the real-time visualization.

98. The system of claim 97, wherein the portion of the real-time visualization is automatically zoomed when the intravascular device is within a threshold distance of a region of interest.

99. The system of claim 98, wherein the portion of the real-time visualization corresponds to the area of ​​interest.

100. The system of claim 99, wherein the region of interest is a treatment area or the location of the intravascular device.

101. The system of claim 100, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

102. The system of claim 98, wherein determining the threshold distance comprises at least one of: determining the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined.

103. The system of claim 102, wherein the spatial distance is a Euclidean distance or a geodesic distance.

104. The system of claim 97, wherein automatically scaling comprises: sorting pixel values ​​of the real-time visualization by their intensities; Normalizing pixel intensity values ​​below a predetermined threshold; as well as A median filter is applied to the normalized pixel intensity values ​​of the real-time visualization.

105. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving at least one first extraluminal image; receiving a second extraluminal image captured during delivery of the intravascular device; detecting movement features in the at least one first extraluminal image and the second extraluminal image; correlating the at least one first extraluminal image and the second extraluminal image based on the detected movement characteristics; as well as A real-time visualization of a location of the intravascular device on one of the at least one first extraluminal image or the second extraluminal image including the intravascular device is provided for output.

106. The non-transitory computer readable medium of claim 105, wherein the at least one first extraluminal image is a high-dose contrast X-ray angiogram.

107. The non-transitory computer readable medium of claim 105 or 106, wherein the second extraluminal image is a low-dose contrast X-ray angiogram.

108. The non-transitory computer-readable medium of any one of claims 105 to 107, wherein the intravascular device is at least one of a stent delivery device, a balloon device, an intravascular imaging probe, a vessel preparation device, or a pressure line.

109. The non-transitory computer-readable medium of any one of claims 105 to 108, wherein the one or more processors are further configured to generate a vessel map based on the at least one first extraluminal image.

110. The non-transitory computer readable medium of any one of claims 105 to 109, wherein the one or more processors are further configured to automatically detect a working vessel using an artificial intelligence (AI) model.

111. The non-transitory computer readable medium of claim 110, wherein when training the AI ​​model to automatically detect the working vessel, the one or more processors are further configured to: annotating the pre-contrast extraluminal image as a line following a trajectory of at least one of the guide wire or the guide catheter; annotating the path of the working blood vessel in the post-contrast extraluminal image; providing the AI ​​model with the annotated pre-contrast extraluminal images and the labeled post-contrast extraluminal images as training data; and The AI ​​model is trained to predict working vessel trajectories.

112. The non-transitory computer readable medium of claim 110 or 111, wherein the one or more processors are further configured to: receiving at least one pre-contrast extraluminal image as input into the AI ​​model; The AI ​​model is trained by enhancing a high-dose extraluminal image into a low-dose extraluminal image as input to segment at least one of a guide catheter, a guidewire, a stent marker, or a balloon marker on the low-dose extraluminal image; detecting a guidewire of the intravascular device by executing the AI ​​model and based on the at least one pre-contrast extraluminal image; by executing the AI ​​model and propagating wire information on a frame-by-frame basis based on detected wires; as well as The working vessel trajectory is automatically predicted by executing the AI ​​model based on the propagated guidewire information.

113. The non-transitory computer-readable medium of any one of claims 105 to 112, wherein the movement feature comprises at least one of: a guide catheter tip, a distal endpoint of a working vessel, optical flow at the guide catheter tip, or optical flow at a distal endpoint of the working vessel.

114. The non-transitory computer readable medium of claim 113, wherein when the movement feature is detected, the one or more processors are further configured to automatically detect the working vessel by executing an artificial intelligence (AI) model.

115. The non-transitory computer-readable medium of any one of claims 105 to 114, wherein when correlating the at least one first extraluminal image with the second extraluminal image, the one or more processors are further configured to: Determine vascular grade migration; Determine cable tip level movement; and Determine the pixel-level movement of blood vessels.

116. The non-transitory computer readable medium of claim 115, wherein when determining the vessel level movement, the one or more processors are further configured to determine a two-dimensional translation vector.

117. The non-transitory computer-readable medium of claim 116, wherein the two-dimensional translation vector corresponds to a two-dimensional translation at an nth frame relative to a first frame.

118. The non-transitory computer readable medium of claim 115, wherein when determining the cable tip level movement, the one or more processors are further configured to determine absolute spatial information and relative spatial information.

119. The non-transitory computer-readable medium of claim 118, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

120. The non-transitory computer readable medium of claim 115, wherein when determining vessel pixel level movement, the one or more processors are further configured to determine absolute spatial information and relative spatial information for a plurality of points on the working vessel.

121. The non-transitory computer-readable medium of claim 120, wherein the absolute spatial information corresponds to coordinates and the relative spatial information corresponds to optical flow between adjacent image frames.

122. The non-transitory computer-readable medium of any one of claims 105 to 121, wherein the one or more processors are further configured to determine a heart cycle of the patient based on the detected movement characteristics.

123. A non-transitory computer-readable medium according to any one of claims 105 to 122, wherein the one or more processors are further configured to determine a spatiotemporal phase match between at least one of the first extraluminal images and at least one of the second extraluminal images based on the detected movement characteristics.

124. The non-transitory computer readable medium of claim 123, wherein the one or more processors are further configured to: resampling detected motion features at a common frame rate; determining a maximum correlation coefficient for pairs of moving features in the at least one first extraluminal image and the second extraluminal image; and A time shift is determined based on the maximum correlation coefficient.

125. The non-transitory computer readable medium of claim 124, wherein the one or more processors are further configured to: determining a drift between movement features detected in the at least one first extraluminal image; and The time shift is adjusted based on the determined drift.

126. The non-transitory computer readable medium of claim 124 or 125, wherein the one or more processors are further configured to: iteratively predicting the time shift; and The time shift is updated to a corrected time shift based on the iteratively predicted time shift.

127. The non-transitory computer-readable medium of any one of claims 124 to 126, wherein the one or more processors are further configured to adjust the spatiotemporal phase matching.

128. The non-transitory computer readable medium of claim 127, wherein when adjusting the spatiotemporal phase matching, the one or more processors are further configured to: identifying another first extraluminal image different from the at least one first extraluminal image of the first extraluminal images; and The spatiotemporal phase matching is adjusted based on the another first extraluminal image.

129. The non-transitory computer readable medium of claim 127 or 128, wherein the one or more processors are further configured to update the real-time visualization based on the adjustment to include the another first extraluminal image.

130. The non-transitory computer readable medium of claim 103, wherein the one or more processors are further configured to detect the intravascular device.

131. The non-transitory computer-readable medium of claim 128, wherein when the intravascular device is detected in the second extraluminal image, the one or more processors are further configured to execute an artificial intelligence (AI) model.

132. The non-transitory computer-readable medium of claim 131 , wherein the one or more processors are further configured to train the AI ​​model, wherein when training the AI ​​model, the one or more processors are further configured to: providing a co-registered dataset as input to the AI ​​model, the co-registered dataset comprising a plurality of intraluminal images and extraluminal images, wherein the plurality of intraluminal images and extraluminal images are annotated images; and The AI ​​model is trained to predict the location of the intravascular device.

133. The non-transitory computer-readable medium of claim 132, wherein the annotated image includes annotations identifying one or more intravascular device markers.

134. The non-transitory computer readable medium of claim 129, wherein the one or more processors are further configured to: detecting an optical flow of the intravascular device; determining a position of the intravascular device in the first frame of the second extraluminal image based on the detected optical flow; and A position of the intravascular device in a subsequent frame of the second extraluminal image is predicted based on the detected optical flow.

135. The non-transitory computer-readable medium of any one of claims 105 to 134, wherein the one or more processors are further configured to provide a treatment zone on at least one of the second extraluminal images or the at least one first extraluminal image for output.

136. The non-transitory computer-readable medium of claim 135, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

137. The non-transitory computer readable medium of claim 136, wherein: The lesion-related region is at least one of a calcification frame, a lipid frame, or an anatomical frame, and The lesion-related region is identified by the one or more processors from another imaging modality and co-registered to the first extraluminal image.

138. The non-transitory computer readable medium of claim 137, wherein the another imaging modality is an intravascular imaging modality.

139. The non-transitory computer-readable medium of any one of claims 105 to 138, wherein the one or more processors are further configured to receive annotations of the at least one first extraluminal image or annotations of the second extraluminal image.

140. The non-transitory computer readable medium of claim 139, wherein upon receiving an annotation, the one or more processors are further configured to: receiving one or more inputs from a user corresponding to the annotation; or Annotations are automatically determined based on vascular data.

141. A non-transitory computer-readable medium according to claim 139 or 140, wherein the annotation includes one or more of the following: plaque burden, fractional flow reserve ("FFR") measurement at one or more locations along the vessel, calcium angle, EEL detection, calcium detection, proximal frame, distal frame, EEL-based indicators, stent decision, score, debulking recommendation, subsequent surgical recommendation, stent placement area, treatment device landing area, balloon device area, vessel preparation device area, or lesion-related area.

142. The non-transitory computer-readable medium of any one of claims 139 to 141, wherein the one or more processors are further configured to update a second of the at least one first extraluminal images or a second of the second extraluminal images based on the received annotation.

143. The non-transitory computer-readable medium of any one of claims 139 to 142, wherein the one or more processors are further configured to automatically capture a screenshot of a real-time visualization of the position of the intravascular device.

144. The non-transitory computer-readable medium of claim 143, wherein the screenshot is automatically captured when the intravascular device is within a threshold distance of an area of ​​interest.

145. The non-transitory computer readable medium of claim 144, wherein the region of interest is a treatment area.

146. The non-transitory computer readable medium of claim 145, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

147. The non-transitory computer-readable medium of claim 144, wherein determining the threshold distance comprises at least one of: determining the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined.

148. The non-transitory computer-readable medium of claim 147, wherein the spatial distance is a Euclidean distance or a geodesic distance.

149. The non-transitory computer-readable medium of any one of claims 105 to 148, wherein the one or more processors are further configured to automatically scale a portion of the real-time visualization.

150. The non-transitory computer-readable medium of claim 149, wherein the portion of the real-time visualization is automatically zoomed when the intravascular device is within a threshold distance of a region of interest.

151. The non-transitory computer-readable medium of claim 150, wherein the portion of the real-time visualization corresponds to the area of ​​interest.

152. The non-transitory computer readable medium of claim 151, wherein the region of interest is a treatment area or a location of the intravascular device.

153. The non-transitory computer-readable medium of claim 152, wherein the treatment zone is at least one of a treatment device landing zone, a balloon device zone, a vascular preparation device zone, or a lesion-related zone.

154. The non-transitory computer-readable medium of claim 150, wherein determining the threshold distance comprises at least one of: determining the number of pixels between an outer boundary of the region of interest and at least one detected marker on the intravascular device, or A spatial distance between the at least one detected marker on the intravascular device and the region of interest is determined.

155. The non-transitory computer-readable medium of claim 154, wherein the spatial distance is a Euclidean distance or a geodesic distance.

156. The non-transitory computer readable medium of claim 143, wherein the one or more processors are further configured to automatically scale, the automatic scaling comprising: sorting pixel values ​​of the real-time visualization by their intensities; Normalizing pixel intensity values ​​below a predetermined threshold; as well as A median filter is applied to the normalized pixel intensity values ​​of the real-time visualization.

157. A method comprising: receiving, by one or more processors, extraluminal images captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detecting, by one or more processors, a plurality of device marker candidates, wherein at least one of the device marker candidates corresponds to the radiopaque marker of the intravascular device; automatically detecting, by the one or more processors using an artificial intelligence (AI) model, a working blood vessel using a plurality of virtual boxes and at least one device marker candidate among the plurality of device marker candidates, wherein at least one of the plurality of virtual boxes includes the at least one device tag candidate among the plurality of device tag candidates; as well as At least one of the plurality of virtual boxes that includes a region of interest of the working vessel is selected, by one or more processors, the region of interest containing the at least one of the plurality of device markers.

158. The method of claim 157, comprising: predicting, by one or more of the processors, a center point of each of the plurality of virtual boxes, Pairing, by one or more processors, the detected device tag candidate with a closest virtual box among the multiple virtual boxes; filtering, by one or more processors, device tag candidates beyond boundaries of the plurality of virtual boxes; updating, by one or more processors, a center point of each of the plurality of virtual boxes using the filtered device marker candidate points; determining, by one or more processors, displacements of the predicted center point and the updated center point; as well as The plurality of virtual boxes are repositioned, by one or more processors, based on the determined displacements.

159. The method of claim 158, wherein updating the center point comprises using device marker candidates to approximate the center point of the box.

160. The method of any one of claims 157 to 159, wherein more than one device tag candidate is located within at least one of the plurality of boxes.

161. A method according to any one of claims 157 to 160, wherein the AI ​​model is trained using annotations on high-dose X-ray angiograms.

162. The method of any one of claims 157 to 161, wherein the extraluminal image is a live X-ray angiogram or a fluoroscopic image.

163. The method of any one of claims 157 to 162, further comprising tracking, by one or more processors, the region of interest during a percutaneous coronary intervention procedure.

164. The method of any one of claims 157 to 163, wherein one or more processors enhance the region of interest using local contrast stretching by selectively enhancing the local contrast between the detected device marker candidate and the surrounding area.

165. The method of any one of claims 157 to 164, further comprising automatically scaling the region of interest by one or more of the processors, wherein the automatically scaling comprises: sorting, by one or more processors, pixel values ​​of the real-time visualization by their intensities; Normalizing, by one or more processors, pixel intensity values ​​below a predetermined threshold; A median filter is applied by one or more processors to the normalized pixel intensity values ​​for the real-time visualization.

166. A system comprising: One or more processors configured to: receiving an extraluminal image captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detecting a plurality of device marker candidates, wherein at least one of the device marker candidates corresponds to the radiopaque marker of the intravascular device; automatically detecting, using an artificial intelligence (AI) model, a plurality of virtual boxes and at least one device marker candidate working blood vessel from among the plurality of device marker candidates, wherein at least one of the plurality of virtual boxes contains the at least one device tag candidate among the plurality of device tag candidates; and At least one of the plurality of virtual boxes is selected to include a region of interest of the working vessel, the region of interest containing the at least one device marker of the plurality of device markers.

167. The system of claim 166, wherein the one or more processors are further configured to: predicting the center point of each of the plurality of boxes, pairing the detected device tag candidate with the closest box among the multiple boxes; filtering device tag candidates beyond the boundaries of the plurality of virtual boxes; updating a center point of each of the plurality of virtual boxes using the filtered device marker candidates; Determining the displacement of the predicted center point and the updated center point; as well as Based on the determined displacements, the plurality of virtual boxes are repositioned according to the determined displacements.

168. The system of claim 167, wherein updating the center point comprises using the device marker candidates to approximate the center point of the virtual box.

169. The system of any one of claims 166 to 168, wherein more than one device tag candidate is located within at least one of the plurality of virtual boxes.

170. The system of any one of claims 166 to 169, wherein the AI ​​model is trained using annotations on high-dose X-ray angiograms.

171. The system of any one of claims 166 to 170, wherein the extraluminal image is a live X-ray angiogram or a fluoroscopic image.

172. The system of any one of claims 166 to 171, wherein the one or more processors are further configured to track the region of interest during a percutaneous coronary intervention procedure.

173. A method according to any one of claims 166 to 172, wherein the one or more processors are further configured to enhance the region of interest using local contrast stretching by selectively enhancing the local contrast between the detected device marker candidates and the surrounding area.

174. The system of any one of claims 166 to 173, wherein the one or more processors are further configured to automatically zoom the region of interest, wherein the automatic zooming comprises: sorting pixel values ​​of the real-time visualization by their intensities; Normalizing pixel intensity values ​​below a predetermined threshold; A median filter is applied to the normalized pixel intensity values ​​of the real-time visualization.

175. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving an extraluminal image captured during delivery of an intravascular device, wherein the intravascular device has a radiopaque marker; detecting a plurality of device marker candidates, wherein at least one of the device marker candidates corresponds to the radiopaque marker of the intravascular device; automatically detecting a working vessel using an artificial intelligence (AI) model, the working vessel comprising a plurality of virtual boxes and at least one of the plurality of device marker candidates, wherein at least one virtual box among the plurality of virtual boxes contains the at least one of the plurality of device tag candidates; and At least one of the plurality of virtual boxes is selected to include a region of interest of the working vessel, the region of interest containing the at least one of the plurality of device markers.

176. The non-transitory computer readable medium of claim 175, wherein the one or more processors are further configured to: predicting the center point of each of the plurality of boxes, pairing the detected device tag candidate with the closest box among the multiple boxes; filtering device tag candidates that fall outside the box; updating a center point of each of the plurality of boxes using the filtered device marker candidate points; Determining the displacement of the predicted center point and the updated center point; as well as The plurality of boxes are repositioned based on the determined displacements.

177. The non-transitory computer-readable medium of claim 176, wherein updating the center point comprises approximating a center point of the box using the device marker candidates.

178. The non-transitory computer-readable medium of any one of claims 175 to 177, wherein more than one device tag candidate is located within at least one of the plurality of virtual boxes.

179. The non-transitory computer-readable medium of any one of claims 175 to 178, wherein the AI ​​model is trained using annotations on high-dose X-ray angiograms.

180. The non-transitory computer-readable medium of any one of claims 175 to 179, wherein the extraluminal image is a live X-ray angiogram or a fluoroscopic image.

181. The non-transitory computer readable medium of any one of claims 175 to 180, wherein the one or more processors are further configured to track the region of interest during a percutaneous coronary intervention procedure.

182. A non-transitory computer-readable medium according to any one of claims 175 to 181, wherein the one or more processors are further configured to enhance the region of interest using local contrast stretching by selectively enhancing the local contrast between the detected device marker candidates and the surrounding area.

183. The non-transitory computer readable medium of any one of claims 175 to 182, wherein the one or more processors are further configured to automatically zoom the region of interest, wherein the automatic zooming comprises: sorting pixel values ​​of the real-time visualization by their intensities; Normalizing pixel intensity values ​​below a predetermined threshold; A median filter is applied to the normalized pixel intensity values ​​of the real-time visualization.

Images